This is the code repository for the paper A hierarchical semantic segmentation framework for computer-vision-based bridge column damage detection

Overview

Bridge-damage-segmentation

This is the code repository for the paper A hierarchical semantic segmentation framework for computer-vision-based bridge column damage detection submitted to the IC-SHM Challenge 2021. The semantic segmentation framework used in this paper leverages importance sampling, semantic mask, and multi-scale test time augmentation to achieve a 0.836 IoU for scene component segmentation and a 0.467 IoU for concrete damage segmentation on the Tokaido Dataset. The framework was implemented on MMSegmentation using Python.

Highlights

Models used in the framework

Backbones

  • HRNet
  • Swin
  • ResNest

Decoder Heads

  • PSPNet
  • UperNet
  • OCRNet

Performance

The following table reports IoUs for structural component segmentation.

Architecture Slab Beam Column Non-structural Rail Sleeper Average
Ensemble 0.891 0.880 0.859 0.660 0.623 0.701 0.785
Ensemble + Importance sampling 0.915 0.912 0.958 0.669 0.618 0.892 0.827
Ensemble + Importance sampling + Multi-scale TTA 0.924 0.929 0.965 0.681 0.621 0.894 0.836

The following table reports IoUs for damage segmentation of pure texture images.

Architecture Concrete damage Exposed rebar Average
Ensemble 0.356 0.536 0.446
Ensemble + Importance sampling 0.708 0.714 0.711
Ensemble + Importance sampling + Multi-scale TTA 0.698 0.727 0.712

The following table reports IoUs for damage segmentation of real scene images.

Architecture Concrete damage Exposed rebar Average
Ensemble 0.235 0.365 0.300
Ensemble + Importance sampling 0.340 0.557 0.448
Ensemble + Importance sampling + Multi-scale TTA 0.350 0.583 0.467
Ensemble + Importance sampling + Multi-scale TTA + Mask 0.379 0.587 0.483

Environment

The code is developed under the following configurations.

  • Hardware: >= 2 GPUs for training, >= 1 GPU for testing. The script supports sbatch training and testing on computer clusters.
  • Software:
    • System: Ubuntu 16.04.3 LTS
    • CUDA >= 10.1
  • Dependencies:

Usage

Environment

  1. Install conda and create a conda environment

    $ conda create -n open-mmlab
    $ source activate open-mmlab
    $ conda install pytorch=1.6.0 torchvision cudatoolkit=10.1 -c pytorch
  2. Install mmcv-full

    $ pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cu101/torch1.6.0/index.html
  3. Install mmsegmentation

    $ pip install git+https://github.com/open-mmlab/mmsegmentation.git
  4. Install other dependencies

    $ pip install opencv, tqdm, numpy, scipy
  5. Download the Tokaido dataset from IC-SHM Challenge 2021.

Training

  1. Example single model training using multiple GPUs
    $ python3 -m torch.distributed.launch --nproc_per_node=2 --nnodes=2 --master_port=$RANDOM ./apis/train_damage_real.py \
      --nw hrnet \
      --cp $CHECKPOINT_DIR \
      --dr $DATA_ROOT \
      --conf $MODEL_CONFIG \
      --bs 16 \
      --train_split $TRAIN_SPLIT_PATH \
      --val_split $VAL_SPLIT_PATH \
      --width 1920 \
      --height 1080 \
      --distributed \
      --iter 100000 \
      --log_iter 10000 \
      --eval_iter 10000 \
      --checkpoint_iter 10000 \
      --multi_loss \
      --ohem \
      --job_name dmg
  2. Example shell script for preparing the whole dataset and train all models for the whole pipeline.
    $ ./scripts/main_training_script.sh

Evlauation

  1. Eval one model

    $ python3 ./test/test.py \
      --nw hrnet \
      --task single \
      --cp $CONFIG_PATH \
      --dr $DATA_ROOT \
      --split_csv $RAW_CSV_PATH \
      --save_path $OUTPOUT_DIR \
      --img_dir $INPUT_IMG_DIR \
      --ann_dir $INPUT_GT_DIR \
      --split $TEST_SPLIT_PATH \
      --type cmp \
      --width 640 \
      --height 360
  2. Example shell script for testing the whole pipeline and generate the output using the IC-SHM Challenge format.

    $ ./scripts/main_testing_script.sh
  3. Visualization (Add the --cmp flag when visualizing components.)

    $ ./modules/viz_label.py \
      --input $SEG_DIR
      --output $OUTPUT_DIR
      --raw_input $IMG_DIR
      --cmp 

Reference

If you find the code useful, please cite the following paper.

You might also like...
This repository contains the official implementation code of the paper Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis, accepted at EMNLP 2021.
This repository contains the official implementation code of the paper Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis, accepted at EMNLP 2021.

MultiModal-InfoMax This repository contains the official implementation code of the paper Improving Multimodal Fusion with Hierarchical Mutual Informa

A Data Annotation Tool for Semantic Segmentation, Object Detection and Lane Line Detection.(In Development Stage)
A Data Annotation Tool for Semantic Segmentation, Object Detection and Lane Line Detection.(In Development Stage)

Data-Annotation-Tool How to Run this Tool? To run this software, follow the steps: git clone https://github.com/Autonomous-Car-Project/Data-Annotation

Computer vision - fun segmentation experience using classic and deep tools :)
Computer vision - fun segmentation experience using classic and deep tools :)

Computer_Vision_Segmentation_Fun Segmentation of Images and Video. Tools: pytorch Models: Classic model - GrabCut Deep model - Deeplabv3_resnet101 Flo

Siamese-nn-semantic-text-similarity - A repository containing comprehensive Neural Networks based PyTorch implementations for the semantic text similarity task Implementation of self-attention mechanisms for general purpose. Focused on computer vision modules. Ongoing repository.
Implementation of self-attention mechanisms for general purpose. Focused on computer vision modules. Ongoing repository.

Self-attention building blocks for computer vision applications in PyTorch Implementation of self attention mechanisms for computer vision in PyTorch

Repository providing a wide range of self-supervised pretrained models for computer vision tasks.

Hierarchical Pretraining: Research Repository This is a research repository for reproducing the results from the project "Self-supervised pretraining

Face Mask Detection System built with OpenCV, TensorFlow using Computer Vision concepts

Face mask detection Face Mask Detection System built with OpenCV, TensorFlow using Computer Vision concepts in order to detect face masks in static im

This is the first released system towards complex meters` detection and recognition, which is implemented by computer vision techniques.
This is the first released system towards complex meters` detection and recognition, which is implemented by computer vision techniques.

A three-stage detection and recognition pipeline of complex meters in wild This is the first released system towards detection and recognition of comp

Codes of paper
Codes of paper "Unseen Object Amodal Instance Segmentation via Hierarchical Occlusion Modeling"

Unseen Object Amodal Instance Segmentation (UOAIS) Seunghyeok Back, Joosoon Lee, Taewon Kim, Sangjun Noh, Raeyoung Kang, Seongho Bak, Kyoobin Lee This

Owner
Jingxiao Liu
PhD Candidate at Stanford University
Jingxiao Liu
Recall Loss for Semantic Segmentation (This repo implements the paper: Recall Loss for Semantic Segmentation)

Recall Loss for Semantic Segmentation (This repo implements the paper: Recall Loss for Semantic Segmentation) Download Synthia dataset The model uses

null 32 Sep 21, 2022
TorchDistiller - a collection of the open source pytorch code for knowledge distillation, especially for the perception tasks, including semantic segmentation, depth estimation, object detection and instance segmentation.

This project is a collection of the open source pytorch code for knowledge distillation, especially for the perception tasks, including semantic segmentation, depth estimation, object detection and instance segmentation.

yifan liu 147 Dec 3, 2022
Pytorch code for ICRA'21 paper: "Hierarchical Cross-Modal Agent for Robotics Vision-and-Language Navigation"

Hierarchical Cross-Modal Agent for Robotics Vision-and-Language Navigation This repository is the pytorch implementation of our paper: Hierarchical Cr

null 43 Nov 21, 2022
BMW TechOffice MUNICH 148 Dec 21, 2022
PyTorch code for the paper "Complementarity is the King: Multi-modal and Multi-grained Hierarchical Semantic Enhancement Network for Cross-modal Retrieval".

Complementarity is the King: Multi-modal and Multi-grained Hierarchical Semantic Enhancement Network for Cross-modal Retrieval (M2HSE) PyTorch code fo

Xinlei-Pei 6 Dec 23, 2022
LBK 20 Dec 2, 2022
This repository contains the source code of our work on designing efficient CNNs for computer vision

Efficient networks for Computer Vision This repo contains source code of our work on designing efficient networks for different computer vision tasks:

Sachin Mehta 386 Nov 26, 2022
PyTorchCV: A PyTorch-Based Framework for Deep Learning in Computer Vision.

PyTorchCV: A PyTorch-Based Framework for Deep Learning in Computer Vision @misc{CV2018, author = {Donny You ([email protected])}, howpubl

Donny You 40 Sep 14, 2022
A PyTorch-Based Framework for Deep Learning in Computer Vision

TorchCV: A PyTorch-Based Framework for Deep Learning in Computer Vision @misc{you2019torchcv, author = {Ansheng You and Xiangtai Li and Zhen Zhu a

Donny You 2.2k Jan 9, 2023
This repository contains the code for the paper "Hierarchical Motion Understanding via Motion Programs"

Hierarchical Motion Understanding via Motion Programs (CVPR 2021) This repository contains the official implementation of: Hierarchical Motion Underst

Sumith Kulal 40 Dec 5, 2022