Implementation of Segnet, FCN, UNet , PSPNet and other models in Keras.

Divam Gupta

Last update: Jan 5, 2023

Related tags

Deep Learning image-segmentation-keras

Overview

Image Segmentation Keras : Implementation of Segnet, FCN, UNet, PSPNet and other models in Keras.

Implementation of various Deep Image Segmentation models in keras.

Link to the full blog post with tutorial : https://divamgupta.com/image-segmentation/2019/06/06/deep-learning-semantic-segmentation-keras.html

Working Google Colab Examples:

Python Interface: https://colab.research.google.com/drive/1q_eCYEzKxixpCKH1YDsLnsvgxl92ORcv?usp=sharing
CLI Interface: https://colab.research.google.com/drive/1Kpy4QGFZ2ZHm69mPfkmLSUes8kj6Bjyi?usp=sharing

Our Other Repositories

Attention based Language Translation in Keras
Ladder Network in Keras model achives 98% test accuracy on MNIST with just 100 labeled examples

Top Contributors

Models

Following models are supported:

model_name	Base Model	Segmentation Model
fcn_8	Vanilla CNN	FCN8
fcn_32	Vanilla CNN	FCN8
fcn_8_vgg	VGG 16	FCN8
fcn_32_vgg	VGG 16	FCN32
fcn_8_resnet50	Resnet-50	FCN32
fcn_32_resnet50	Resnet-50	FCN32
fcn_8_mobilenet	MobileNet	FCN32
fcn_32_mobilenet	MobileNet	FCN32
pspnet	Vanilla CNN	PSPNet
vgg_pspnet	VGG 16	PSPNet
resnet50_pspnet	Resnet-50	PSPNet
unet_mini	Vanilla Mini CNN	U-Net
unet	Vanilla CNN	U-Net
vgg_unet	VGG 16	U-Net
resnet50_unet	Resnet-50	U-Net
mobilenet_unet	MobileNet	U-Net
segnet	Vanilla CNN	Segnet
vgg_segnet	VGG 16	Segnet
resnet50_segnet	Resnet-50	Segnet
mobilenet_segnet	MobileNet	Segnet

Example results for the pre-trained models provided :

Input Image	Output Segmentation Image

Getting Started

Prerequisites

Keras ( recommended version : 2.4.3 )
OpenCV for Python
Tensorflow ( recommended version : 2.4.1 )

apt-get install -y libsm6 libxext6 libxrender-dev
pip install opencv-python

Installing

Install the module

Recommended way:

pip install --upgrade git+https://github.com/divamgupta/image-segmentation-keras

or

pip install keras-segmentation

or

git clone https://github.com/divamgupta/image-segmentation-keras
cd image-segmentation-keras
python setup.py install

Pre-trained models:

from keras_segmentation.pretrained import pspnet_50_ADE_20K , pspnet_101_cityscapes, pspnet_101_voc12

model = pspnet_50_ADE_20K() # load the pretrained model trained on ADE20k dataset

model = pspnet_101_cityscapes() # load the pretrained model trained on Cityscapes dataset

model = pspnet_101_voc12() # load the pretrained model trained on Pascal VOC 2012 dataset

# load any of the 3 pretrained models

out = model.predict_segmentation(
    inp="input_image.jpg",
    out_fname="out.png"
)

Preparing the data for training

You need to make two folders

Images Folder - For all the training images
Annotations Folder - For the corresponding ground truth segmentation images

The filenames of the annotation images should be same as the filenames of the RGB images.

The size of the annotation image for the corresponding RGB image should be same.

For each pixel in the RGB image, the class label of that pixel in the annotation image would be the value of the blue pixel.

Example code to generate annotation images :

import cv2
import numpy as np

ann_img = np.zeros((30,30,3)).astype('uint8')
ann_img[ 3 , 4 ] = 1 # this would set the label of pixel 3,4 as 1

cv2.imwrite( "ann_1.png" ,ann_img )

Only use bmp or png format for the annotation images.

Download the sample prepared dataset

Download and extract the following:

https://drive.google.com/file/d/0B0d9ZiqAgFkiOHR1NTJhWVJMNEU/view?usp=sharing

You will get a folder named dataset1/

Using the python module

You can import keras_segmentation in your python script and use the API

from keras_segmentation.models.unet import vgg_unet

model = vgg_unet(n_classes=51 ,  input_height=416, input_width=608  )

model.train(
    train_images =  "dataset1/images_prepped_train/",
    train_annotations = "dataset1/annotations_prepped_train/",
    checkpoints_path = "/tmp/vgg_unet_1" , epochs=5
)

out = model.predict_segmentation(
    inp="dataset1/images_prepped_test/0016E5_07965.png",
    out_fname="/tmp/out.png"
)

import matplotlib.pyplot as plt
plt.imshow(out)

# evaluating the model 
print(model.evaluate_segmentation( inp_images_dir="dataset1/images_prepped_test/"  , annotations_dir="dataset1/annotations_prepped_test/" ) )

Usage via command line

You can also use the tool just using command line

Visualizing the prepared data

You can also visualize your prepared annotations for verification of the prepared data.

python -m keras_segmentation verify_dataset \
 --images_path="dataset1/images_prepped_train/" \
 --segs_path="dataset1/annotations_prepped_train/"  \
 --n_classes=50

python -m keras_segmentation visualize_dataset \
 --images_path="dataset1/images_prepped_train/" \
 --segs_path="dataset1/annotations_prepped_train/"  \
 --n_classes=50

Training the Model

To train the model run the following command:

python -m keras_segmentation train \
 --checkpoints_path="path_to_checkpoints" \
 --train_images="dataset1/images_prepped_train/" \
 --train_annotations="dataset1/annotations_prepped_train/" \
 --val_images="dataset1/images_prepped_test/" \
 --val_annotations="dataset1/annotations_prepped_test/" \
 --n_classes=50 \
 --input_height=320 \
 --input_width=640 \
 --model_name="vgg_unet"

Choose model_name from the table above

Getting the predictions

To get the predictions of a trained model

python -m keras_segmentation predict \
 --checkpoints_path="path_to_checkpoints" \
 --input_path="dataset1/images_prepped_test/" \
 --output_path="path_to_predictions"

Video inference

To get predictions of a video

python -m keras_segmentation predict_video \
 --checkpoints_path="path_to_checkpoints" \
 --input="path_to_video" \
 --output_file="path_for_save_inferenced_video" \
 --display

If you want to make predictions on your webcam, don't use --input, or pass your device number: --input 0
--display opens a window with the predicted video. Remove this argument when using a headless system.

Model Evaluation

To get the IoU scores

python -m keras_segmentation evaluate_model \
 --checkpoints_path="path_to_checkpoints" \
 --images_path="dataset1/images_prepped_test/" \
 --segs_path="dataset1/annotations_prepped_test/"

Fine-tuning from existing segmentation model

The following example shows how to fine-tune a model with 10 classes .

from keras_segmentation.models.model_utils import transfer_weights
from keras_segmentation.pretrained import pspnet_50_ADE_20K
from keras_segmentation.models.pspnet import pspnet_50

pretrained_model = pspnet_50_ADE_20K()

new_model = pspnet_50( n_classes=51 )

transfer_weights( pretrained_model , new_model  ) # transfer weights from pre-trained model to your model

new_model.train(
    train_images =  "dataset1/images_prepped_train/",
    train_annotations = "dataset1/annotations_prepped_train/",
    checkpoints_path = "/tmp/vgg_unet_1" , epochs=5
)

Knowledge distillation for compressing the model

The following example shows transfer the knowledge from a larger ( and more accurate ) model to a smaller model. In most cases the smaller model trained via knowledge distilation is more accurate compared to the same model trained using vanilla supervised learning.

from keras_segmentation.predict import model_from_checkpoint_path
from keras_segmentation.models.unet import unet_mini
from keras_segmentation.model_compression import perform_distilation

model_large = model_from_checkpoint_path( "/checkpoints/path/of/trained/model" )
model_small = unet_mini( n_classes=51, input_height=300, input_width=400  )

perform_distilation ( data_path="/path/to/large_image_set/" , checkpoints_path="path/to/save/checkpoints" , 
    teacher_model=model_large ,  student_model=model_small  , distilation_loss='kl' , feats_distilation_loss='pa' )

Adding custom augmentation function to training

The following example shows how to define a custom augmentation function for training.

from keras_segmentation.models.unet import vgg_unet
from imgaug import augmenters as iaa

def custom_augmentation():
    return  iaa.Sequential(
        [
            # apply the following augmenters to most images
            iaa.Fliplr(0.5),  # horizontally flip 50% of all images
            iaa.Flipud(0.5), # horizontally flip 50% of all images
        ])

model = vgg_unet(n_classes=51 ,  input_height=416, input_width=608)

model.train(
    train_images =  "dataset1/images_prepped_train/",
    train_annotations = "dataset1/annotations_prepped_train/",
    checkpoints_path = "/tmp/vgg_unet_1" , epochs=5, 
    do_augment=True, # enable augmentation 
    custom_augmentation=custom_augmentation # sets the augmention function to use
)

Custom number of input channels

The following example shows how to set the number of input channels.

from keras_segmentation.models.unet import vgg_unet

model = vgg_unet(n_classes=51 ,  input_height=416, input_width=608, 
                 channels=1 # Sets the number of input channels
                 )

model.train(
    train_images =  "dataset1/images_prepped_train/",
    train_annotations = "dataset1/annotations_prepped_train/",
    checkpoints_path = "/tmp/vgg_unet_1" , epochs=5, 
    read_image_type=0  # Sets how opencv will read the images
                       # cv2.IMREAD_COLOR = 1 (rgb),
                       # cv2.IMREAD_GRAYSCALE = 0,
                       # cv2.IMREAD_UNCHANGED = -1 (4 channels like RGBA)
)

Custom preprocessing

The following example shows how to set a custom image preprocessing function.

from keras_segmentation.models.unet import vgg_unet

def image_preprocessing(image):
    return image + 1

model = vgg_unet(n_classes=51 ,  input_height=416, input_width=608)

model.train(
    train_images =  "dataset1/images_prepped_train/",
    train_annotations = "dataset1/annotations_prepped_train/",
    checkpoints_path = "/tmp/vgg_unet_1" , epochs=5,
    preprocessing=image_preprocessing # Sets the preprocessing function
)

Custom callbacks

The following example shows how to set custom callbacks for the model training.

from keras_segmentation.models.unet import vgg_unet
from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping

model = vgg_unet(n_classes=51 ,  input_height=416, input_width=608 )

# When using custom callbacks, the default checkpoint saver is removed
callbacks = [
    ModelCheckpoint(
                filepath="checkpoints/" + model.name + ".{epoch:05d}",
                save_weights_only=True,
                verbose=True
            ),
    EarlyStopping()
]

model.train(
    train_images =  "dataset1/images_prepped_train/",
    train_annotations = "dataset1/annotations_prepped_train/",
    checkpoints_path = "/tmp/vgg_unet_1" , epochs=5,
    callbacks=callbacks
)

Multi input image input

The following example shows how to add additional image inputs for models.

from keras_segmentation.models.unet import vgg_unet

model = vgg_unet(n_classes=51 ,  input_height=416, input_width=608)

model.train(
    train_images =  "dataset1/images_prepped_train/",
    train_annotations = "dataset1/annotations_prepped_train/",
    checkpoints_path = "/tmp/vgg_unet_1" , epochs=5,
    other_inputs_paths=[
        "/path/to/other/directory"
    ],
    
    
#     Ability to add preprocessing
    preprocessing=[lambda x: x+1, lambda x: x+2, lambda x: x+3], # Different prepocessing for each input
#     OR
    preprocessing=lambda x: x+1, # Same preprocessing for each input
)

Projects using keras-segmentation

Here are a few projects which are using our library :

If you use our code in a publicly available project, please add the link here ( by posting an issue or creating a PR )

Comments

Checkpoint is not found
I am trying to run this command after I have trained the network but it is giving an error.

python -m keras_segmentation predict \ --checkpoints_path="path_to_checkpoints" \ --input_path="dataset1/images_prepped_test/" \ --output_path="path_to_predictions"

File "C:\Users\Caiow\AppData\Local\Programs\Python\Python38\lib\site-packages\keras_segmentation\predict.py", line 175, in predict_multiple model = model_from_checkpoint_path(checkpoints_path) File "C:\Users\Caiow\AppData\Local\Programs\Python\Python38\lib\site-packages\keras_segmentation\predict.py", line 29, in model_from_checkpoint_path assert (latest_weights is not None), "Checkpoint not found." AssertionError: Checkpoint not found.
opened by Caioww 23
vgg.load_weights(VGG_Weights_path) Error: input shapes: [102400,4096], [25088,4096]

I am getting an error at the line: vgg.load_weights(VGG_Weights_path)

the error is: ValueError: Dimension 0 in both shapes must be equal, but are 102400 and 25088 for 'Assign_26' (op: 'Assign') with input shapes: [102400,4096], [25088,4096].

Please note that: I am running Keras with the Tensorflow backend because, with the Theano backend, I am getting an error which causes Theano to run on the CPU rather thsan the GPU (and only on 1 Core of the CPU). ERROR (theano.sandbox.cuda): Failed to compile cuda_ndarray.cu: ('nvcc return status', 2, ....

opened by VeniVidiGavi 15
Problem when loading weights

Hi, I was following this tutorial

https://divamgupta.com/image-segmentation/2019/06/06/deep-learning-semantic-segmentation-keras.html

And everrything is great, but when I try to use:

from keras_segmentation import predict

predict( checkpoints_path="checkpoints/vgg_unet_1", inp="dataset_path/images_prepped_test/0016E5_07965.png", out_fname="output.png" )

it says:

TypeError: 'module' object is not callable

The same happens with:

from keras_segmentation import predict_multiple

predict_multiple( checkpoints_path="checkpoints/vgg_unet_1", inp_dir="dataset_path/images_prepped_test/", out_dir="outputs/" )

cannot import name 'predict_multiple'

cannot figure out what the problem is, any clues?

opened by eiraola 14

ModuleNotFoundError: No module named 'VGGUnet'

When training the model getting following error.

Traceback (most recent call last):
  File "train.py", line 2, in <module>
    import Models , LoadBatches
  File "/home/nd/image-segmentation-keras-master/Models/__init__.py", line 1, in <module>
    import VGGUnet
ModuleNotFoundError: No module named 'VGGUnet'

How to solve it.

opened by hiteshnitetc 14

Bad prediction even with high training and validation accuracy

hey guys,

i am using this code for image segmentation:

`from keras_segmentation.models.unet import unet

model = unet(n_classes=3) model.train(n_classes=3, train_images = "/content/Training/images", train_annotations = "/content/Training/labels", checkpoints_path = "/content/checkpoints" , epochs=1,validate=True,val_images="/content/Test/images",val_annotations="/content/Test/labels") ` Epoch 1/1 512/512 [==============================] - 252s 492ms/step - loss: 0.1199 - accuracy: 0.9728 - val_loss: 7.3933e-05 - val_accuracy: 1.0000 saved /content/checkpoints.model.0 Finished Epoch 0

I have high accuracies but bad prediction. I tried different models. All give me the same result.

Image Label Prediction

opened by Alsen57 13
Assertion error at LoadBatches.py for training

I'm having the following error: /LoadBatches.py", line 71, in imageSegmentationGenerator assert( im.split('/')[-1].split(".")[0] == seg.split('/')[-1].split(".")[0] ) AssertionError

Any thoughts? =(

opened by matheuscass 13
Adds customization options to the segmentation models and other improvements

Main additions: Adds the ability to add custom callbacks to the model training sequences Adds the ability to add custom augmentation functions to the model training Adds the ability to add multi-image input with data augmentation to the models easily

Minor additions: Uses TensorFlow model checkpoint instead of custom Keras one. Uses tensorflow tf.train.latest_checkpoint function instead of custom find_latest_checkpoint function Updates some of the deprecated augmentation function classes (using imgaug>=0.4.0) For the auto-resume checkpoint boolean in training makes it so that it continues from the last checkpoint epoch number instead from 0 again Uses the model.fit function instead of the model.fit_generator function If the checkpoint base folder does not exist, create it Updates the dataset visualization part of the library to accommodate different augmentation functions and image size

opened by Marius-Juston 9
AssertionError: Checkpoint not found. analogous to #237 issue

Hi,

I've encountered the same issue of #237. I tried to run this google colab notebook, without positive results

https://colab.research.google.com/drive/1Kpy4QGFZ2ZHm69mPfkmLSUes8kj6Bjyi?usp=sharing#scrollTo=79ib1d3xFpAy

You developed a fantastic library. I hope you will fix this issue. Thanks, Domenico

opened by DomenicoMessina 8
video prediction function callable from CLI

Takes a local video or webcam stream, obtains the inference mask, and overlays it over original frame for better representation.

I tried to follow your repo code style, but there is code duplicity between this function and predict().

It expects weights path, the input video source, and, if desired, the frame speed.

Fixes #83

Thanks for this amazing repo and hope it helps.

opened by JaledMC 7

ImportError: cannot import name 'tf' from 'keras.backend'

I am having problems to execute the pre-trained models:

import keras_segmentation

model = keras_segmentation.pretrained.pspnet_50_ADE_20K() 
out = model.predict_segmentation(
    inp="input_image.jpg",
    out_fname="out.png"
)

ImportError                               Traceback (most recent call last)
<ipython-input-22-1a69a6bbd448> in <module>
      1 print(keras.__version__)
----> 2 model = keras_segmentation.pretrained.pspnet_50_ADE_20K()

~/anaconda3/lib/python3.7/site-packages/keras_segmentation/pretrained.py in pspnet_50_ADE_20K()
     44     latest_weights =  keras.utils.get_file( "pspnet50_ade20k.h5" , model_url  )
     45 
---> 46     return model_from_checkpoint_path( model_config , latest_weights  )
     47 
     48 

~/anaconda3/lib/python3.7/site-packages/keras_segmentation/pretrained.py in model_from_checkpoint_path(model_config, latest_weights)
      7 def model_from_checkpoint_path( model_config , latest_weights  ):
      8 
----> 9         model = model_from_name[ model_config['model_class']  ]( model_config['n_classes'] , input_height=model_config['input_height'] , input_width=model_config['input_width'] )
     10         model.load_weights(latest_weights)
     11         return model

~/anaconda3/lib/python3.7/site-packages/keras_segmentation/models/pspnet.py in pspnet_50(n_classes, input_height, input_width)
    103 
    104 def pspnet_50( n_classes ,  input_height=473, input_width=473 ):
--> 105     from ._pspnet_2 import _build_pspnet
    106 
    107     nb_classes = n_classes

~/anaconda3/lib/python3.7/site-packages/keras_segmentation/models/_pspnet_2.py in <module>
     10 from keras.optimizers import SGD
     11 
---> 12 from keras.backend import tf as ktf
     13 import tensorflow as tf
     14 

ImportError: cannot import name 'tf' from 'keras.backend' (/home/alex/anaconda3/lib/python3.7/site-packages/keras/backend/__init__.py)

opened by alexst07 6

cannot import name transfer_weights

When i tried to fine tune a model with 10 classes I am getting this error

from keras_segmentation.models.model_utils import transfer_weights

ImportError: cannot import name transfer_weights

Please help me fix this issue asap.

opened by fizaict 6
Custom Augmentation: AttributeError: 'Compose' object has no attribute 'to_deterministic'

Hi, I've been really impressed by all of your work so far! It has been working great for me except for this one error. When I try to create a custom augmentation for my images and masks, I keep receiving an error saying AttributeError: 'Compose' object has no attribute 'to_deterministic'. I'm not sure what the issue is. Any help would be greatly appreciated!

from imgaug import augmenters as iaa

def custom_augmentation(): return iaa.Sequential( [ # apply the following augmenters to most images iaa.Fliplr(0.5), # horizontally flip 50% of all images iaa.Flipud(0.5), # horizontally flip 50% of all images ])

model.train( train_images = images_train_full_dir, train_annotations = annotations_train_full_dir, val_images = images_prepped_val_dir, val_annotations = annotations_prepped_val_dir, validate = True, checkpoints_path = "/tmp/vgg_unet_1" , steps_per_epoch = 25, batch_size = 8, preprocessing = normalization, epochs=5, callbacks = callbacks, do_augment=True, # enable augmentation custom_augmentation=custom_augmentation )

opened by jjohn485 0
Wrong number of classes in visualize_segmentation

Hello there, I think I have spotted a small mistake in the code.

TL;DR:

n_classes should be np.max(seg_arr) + 1 and not np.max(seg_arr)

How to see the bug:

Visualize an image without specifying the class number: Observe a color on the image that is not in the legend (black). It is 'normal' because in get_colored_segmentation_image, not found classes number are set to [0, 0, 0]. Once the class number is correctly set (does not use the default code) then it works as expected.

Where is it?

File: https://github.com/divamgupta/image-segmentation-keras/blob/master/keras_segmentation/predict.py Function: visualize_segmentation Line: 104 Found: n_classes = np.max(seg_arr) Should: n_classes = np.max(seg_arr) + 1

Hope it will help Cheers!

opened by gaetanmuck 0
Local Machine Multi GPU usage

Hi there. I am at the newest version and using GPU training , libraries are installed as recommended.

Yet the libarary is still only using one of my 2 GPUs.

Am I missing something? Do I need to enable it through a special parameter?

opened by GGDRriedel 0
Compatibility with TensorFlow 2.4.1
Under TF 2, the output for the first checkpoint is .00001.index and .00001.data-00000-of-00001 rather than .0. get_epoch_number_from_path now strips path using os.path.basename and the .index suffix to properly return the number of the checkpoint.

model_from_checkpoint_path now uses os.path.join to avoid having to supply a trailing slash for the model directory.
opened by fracpete 0