Semantic Segmentation for Aerial Imagery using Convolutional Neural Network

Shunta Saito

Last update: Sep 23, 2022

Related tags

Deep Learning ssai

Overview

This repo has been deprecated because whole things are re-implemented by using Chainer and I did refactoring for many codes. So please check this newer version: https://github.com/mitmul/ssai-cnn

Semantic Segmentation for Aerial Imagery

Extract building and road from aerial imagery

Requirements

OpenCV 2.4.10
Boost 1.57.0
Boost.NumPy
Caffe (modified caffe: https://github.com/mitmul/caffe)
- NOTE: Build the ssai branch of the above repository

Data preparation

$ bash shells/donwload.sh
$ python scripts/create_dataset.py --dataset multi
$ python scripts/create_dataset.py --dataset single
$ python scripts/create_dataset.py --dataset roads_mini
$ python scripts/create_dataset.py --dataset roads
$ python scripts/create_dataset.py --dataset buildings
$ python scripts/create_dataset.py --dataset merged

Massatusetts Building & Road dataset

mass_roads
- train: 8458173 patches
  - epoch: 66079 mini-batches (mini-batch size: 128)
- valid: 126281 patches
  - epoch: 987 mini-batches (mini-batch size: 128)
- test: 440932 patches
  - epoch: 3445 mini-batches (mini-batch size: 128)
mass_roads_mini, mass_buildings, mass_merged
- train: 1119872 patches
  - epoch: 8749 mini-batches (mini-batch size: 128)
- valid: 36100 patches
  - epoch: 282 mini-batches (mini-batch size: 128)
- test: 89968 patches
  - epoch: 703 mini-batches (mini-batch size: 128)

Create Models

$ python scripts/create_models.py --seed seeds/model_seeds.json --caffe_dir $HOME/lib/caffe/build/install

Start training

$ bash shells/train.sh models/Mnih_CNN

will create a directory named results/Mnih_CNN_{started date}.

Prediction

$ cd results/Mnih_CNN_{started date}
$ python ../../scripts/test_prediction.py --model predict.prototxt --weight snapshots/Mnih_CNN_iter_1000000.caffemodel --img_dir ../../data/mass_merged/test/sat --channel 3

Build Library for Evaluation

$ cd lib
$ mkdir build
$ cd build
$ cmake ../
$ make

Evaluation

$ cd results/Mnih_CNN_{started date}
$ python ../../scripts/test_evaluation.py --map_dir ../../data/mass_merged/test/map --result_dir prediction_1000000 --channel 3

Model averaging

$ python ../scripts/batch_evaluation.py --offset True
$ mkdir Mnih_CNN_Merged
$ cd Mnih_CNN_Merged
$ python ../../scripts/test_evaluation.py --map_dir ../../data/mass_merged/test/map --result_dir ./prediction_100000 --channel 3 --offset 0 --pad 31

Comments

Issue while creating models even though the project is built

Hi,

I am trying to replicate your work using caffe, but am encountering the following issue when i try to create a model.

The ssai.so has been build and is stored in the following path: /home/ubuntu/src/ssai/lib/build/ssai.so

I am running the following command to create the models: $ python scripts/create_models.py --seed seeds/model_seeds.json --caffe_dir $HOME/src/caffe/

and i get the following stack trace:

ubuntu@ip-172-31-30-95:~/src/ssai$ python scripts/create_models.py --seed seeds/model_seeds.json --caffe_dir $HOME/src/caffe/ Namespace(batch_size=128, caffe_dir='/home/ubuntu/src/caffe/', crop_size=64, dataset_backend='lmdb', seed='seeds/model_seeds.json') /home/ubuntu/src/caffe/python/caffe/pycaffe.py:13: RuntimeWarning: to-Python converter for boost::shared_ptrcaffe::Net already registered; second conversion method ignored. from ._caffe import Net, SGDSolver, NesterovSolver, AdaGradSolver,
/home/ubuntu/src/caffe/python/caffe/pycaffe.py:13: RuntimeWarning: to-Python converter for boost::shared_ptrcaffe::Blob already registered; second conversion method ignored. from ._caffe import Net, SGDSolver, NesterovSolver, AdaGradSolver,
/home/ubuntu/src/caffe/python/caffe/pycaffe.py:13: RuntimeWarning: to-Python converter for boost::shared_ptrcaffe::Solver already registered; second conversion method ignored. from ._caffe import Net, SGDSolver, NesterovSolver, AdaGradSolver,
Traceback (most recent call last): File "/home/ubuntu/src/caffe//python/draw_net.py", line 45, in main() File "/home/ubuntu/src/caffe//python/draw_net.py", line 39, in main text_format.Merge(open(args.input_net_proto_file).read(), net) File "/usr/local/lib/python2.7/dist-packages/google/protobuf/text_format.py", line 265, in Merge return MergeLines(text.split('\n'), message) File "/usr/local/lib/python2.7/dist-packages/google/protobuf/text_format.py", line 298, in MergeLines _ParseOrMerge(lines, message, True) File "/usr/local/lib/python2.7/dist-packages/google/protobuf/text_format.py", line 229, in _ParseOrMerge _MergeField(tokenizer, message, allow_multiple_scalars) File "/usr/local/lib/python2.7/dist-packages/google/protobuf/text_format.py", line 382, in _MergeField _MergeField(tokenizer, sub_message, allow_multiple_scalars) File "/usr/local/lib/python2.7/dist-packages/google/protobuf/text_format.py", line 356, in _MergeField message_descriptor.full_name, name)) google.protobuf.text_format.ParseError: 53:3 : Message type "caffe.LayerParameter" has no field named "patch_transformer_param". Traceback (most recent call last): File "scripts/create_models.py", line 611, in 'models/%s/net.png' % model_name]) File "/usr/lib/python2.7/subprocess.py", line 573, in check_output raise CalledProcessError(retcode, cmd, output=output) subprocess.CalledProcessError: Command '['python', '/home/ubuntu/src/caffe//python/draw_net.py', u'models/VGG_PReLU_Roads/train_test.prototxt', u'models/VGG_PReLU_Roads/net.png']' returned non-zero exit status 1

Note: I was doing this while the data download was in the progress with the bash shell/

Kindly help.

opened by smartinsert 7
"caffe.LayerParameter" has no field named "patch_transformer_param"

Hi, I'm trying to run training model for your scripts. But i'm getting an error. What could be the reason?

Best regards, Ilya

start learning models/Mnih_CNN_Buildings kalinovskiy@kalinovskiy-virtual-machine:/project/ssai-master$ I1013 02:18:28.541991 15748 caffe.cpp:117] Use CPU. I1013 02:18:28.542275 15748 caffe.cpp:121] Starting Optimization I1013 02:18:28.542351 15748 solver.cpp:32] Initializing solver from parameters: test_iter: 100 test_interval: 1000 base_lr: 0.0005 display: 100 max_iter: 400000 lr_policy: "step" gamma: 0.1 momentum: 0.9 weight_decay: 0.0005 stepsize: 100000 snapshot: 10000 snapshot_prefix: "snapshots/Mnih_CNN_Buildings" solver_mode: CPU device_id: 2 random_seed: 1701 net: "train_test.prototxt" solver_type: SGD I1013 02:18:28.542444 15748 solver.cpp:70] Creating training net from net file: train_test.prototxt [libprotobuf ERROR google/protobuf/text_format.cc:274] Error parsing text-format caffe.NetParameter: 53:27: Message type "caffe.LayerParameter" has no field named "patch_transformer_param". F1013 02:18:28.542604 15748 upgrade_proto.cpp:928] Check failed: ReadProtoFromTextFile(param_file, param) Failed to parse NetParameter file: train_test.prototxt *** Check failure stack trace: *** @ 0x7fd8bafc8ea4 (unknown) @ 0x7fd8bafc8deb (unknown) @ 0x7fd8bafc87bf (unknown) @ 0x7fd8bafcba35 (unknown) @ 0x7fd8bb3df941 caffe::ReadNetParamsFromTextFileOrDie() @ 0x7fd8bb4036e2 caffe::Solver<>::InitTrainNet() @ 0x7fd8bb404760 caffe::Solver<>::Init() @ 0x7fd8bb404946 caffe::Solver<>::Solver() @ 0x40dd50 caffe::GetSolver<>() @ 0x406e22 train() @ 0x40472b main @ 0x7fd8ba4c8a40 (unknown) @ 0x404d79 _start @ (nil) (unknown)

opened by Bkmz21 5
Pre-trained Model

Hello @mitmul. First of all, I appreciate sharing this great code and project! For my research, I am trying to extract some features from your trained network. However, I found out that there is no pre-trained model (or saved snapshot) that was trained with your dataset in this repo. Instead of training from scratch with your network definition and dataset, would it be possible to ask for sharing the pre-trained model that you used in these slides (http://www.slideshare.net/mitmul/building-and-road-detection-from-large-aerial-imagery/1)? Thank you for your time!

opened by ghost 3
predict.prototxt fix

The patch_transformer layer from the generated predict.prototxt required image_label for the bottom blob. I believe this shoudn't happen, since no labels are supposed to be required for prediction. The changes below make prediction possible. Am I wrong?

Totally unrelated, could you please add a seed for the Plain-Mnih-NN-Multi-S-Maxout with Dropout model used in your article "Building and road detection from large aerial imagery"? My group is working on the same topic and we would love to compare our results [pixel by pixel, if possible] with your predictions.

Cheers.

opened by onorabil 2
Unable to train the models

Hi @mitmul,

I am trying to train the roads data set using the Mnih_CNN_Roads model created by running the create_model.sh script.

The ssai/shells/train.sh script tries to run $CAFFE_DIR/bin/caffe which is unavailable even though your modified caffe version was succcesfully installed and test using the mnist sample tests.

When i tried pointing the same command to $CAFFE_DIR/distribute/caffe.bin, the program crashed.

This is the modified training command that i am using:

nohup $CAFFE_DIR/distribute/bin/caffe.bin train
-solver=$PWD/solver.prototxt &

This is dumping a core.

Please help to rectify this.

opened by smartinsert 4

Semantic Segmentation for Aerial Imagery using Convolutional Neural Network

Related tags

Overview

This repo has been deprecated because whole things are re-implemented by using Chainer and I did refactoring for many codes. So please check this newer version: https://github.com/mitmul/ssai-cnn

Semantic Segmentation for Aerial Imagery

Requirements

Data preparation

Massatusetts Building & Road dataset

Create Models

Start training

Prediction

Build Library for Evaluation

Evaluation

Model averaging

Comments

Issue while creating models even though the project is built

"caffe.LayerParameter" has no field named "patch_transformer_param"

Pre-trained Model

predict.prototxt fix

Unable to train the models

Owner

Shunta Saito

FactSeg: Foreground Activation Driven Small Object Semantic Segmentation in Large-Scale Remote Sensing Imagery (TGRS)

This is a model made out of Neural Network specifically a Convolutional Neural Network model

An implementation of the research paper "Retina Blood Vessel Segmentation Using A U-Net Based Convolutional Neural Network"

Recall Loss for Semantic Segmentation (This repo implements the paper: Recall Loss for Semantic Segmentation)

Experiments on Flood Segmentation on Sentinel-1 SAR Imagery with Cyclical Pseudo Labeling and Noisy Student Training

PyTorch implementation of ShapeConv: Shape-aware Convolutional Layer for RGB-D Indoor Semantic Segmentation.

Train neural network for semantic segmentation (deep lab V3) with pytorch in less then 50 lines of code

Mae segmentation - Reproduction of semantic segmentation using masked autoencoder (mae)

Scripts for training an AI to play the endless runner Subway Surfers using a supervised machine learning approach by imitation and a convolutional neural network (CNN) for image classification

Speech Separation Using an Asynchronous Fully Recurrent Convolutional Neural Network

Convolutional neural network web app trained to track our infant’s sleep schedule using our Google Nest camera.

Some code of the implements of Geological Modeling Using 3D Pixel-Adaptive and Deformable Convolutional Neural Network

Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentation, CVPR 2018

Code for "FPS-Net: A convolutional fusion network for large-scale LiDAR point cloud segmentation".

A PyTorch implementation for V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation

Siamese-nn-semantic-text-similarity - A repository containing comprehensive Neural Networks based PyTorch implementations for the semantic text similarity task

Build upon neural radiance fields to create a scene-specific implicit 3D semantic representation, Semantic-NeRF

Code and model benchmarks for "SEVIR : A Storm Event Imagery Dataset for Deep Learning Applications in Radar and Satellite Meteorology"

Change is Everywhere: Single-Temporal Supervised Object Change Detection in Remote Sensing Imagery (ICCV 2021)