An Official Repo of CVPR '20 "MSeg: A Composite Dataset for Multi-Domain Segmentation"

Overview

Linux CI

Creative Commons License

This is the code for the paper:

MSeg: A Composite Dataset for Multi-domain Semantic Segmentation (CVPR 2020, Official Repo) [CVPR PDF] [Journal PDF]
John Lambert*, Zhuang Liu*, Ozan Sener, James Hays, Vladlen Koltun
Presented at CVPR 2020. Link to MSeg Video (3min)

NEWS:

  • [Dec. 2021]: An updated journal-length version of our work is now available on ArXiv here.

This repo is the first of 4 repos that introduce our work. It provides utilities to download the MSeg dataset (which is nontrivial), and prepare the data on disk in a unified taxonomy.

Three additional repos are also provided:

  • mseg-semantic: provides HRNet-W48 Training (sufficient to train a winning entry on the WildDash benchmark)
  • mseg-panoptic: provides Panoptic-FPN and Mask-RCNN training, based on Detectron2 (will be introduced in January 2021)
  • mseg-mturk: utilities to perform large-scale Mechanical Turk re-labeling

Install the MSeg module:

  • mseg can be installed as a python package using

      pip install -e /path_to_root_directory_of_the_repo/
    

Make sure that you can run import mseg in python, and you are good to go!

Download MSeg

The MSeg Taxonomy

We provide comprehensive class definitions and examples here. We provide here a master spreadsheet mapping all training datasets to the MSeg Taxonomy, and the MSeg Taxonomy to test datasets. Please consult taxonomy_FAQ.md to learn what each of the dataset taxonomy names means.

Citing MSeg

If you find this code useful for your research, please cite:

@InProceedings{MSeg_2020_CVPR,
author = {Lambert, John and Liu, Zhuang and Sener, Ozan and Hays, James and Koltun, Vladlen},
title = {{MSeg}: A Composite Dataset for Multi-domain Semantic Segmentation},
booktitle = {Computer Vision and Pattern Recognition (CVPR)},
year = {2020}
}

Repo Structure

  • download_scripts: code and instructions to download the entire MSeg dataset
  • mseg: Python module, including
    • dataset_apis
    • dataset_lists: ordered classnames for each dataset, and corresponding relative rgb/label file paths
    • label_preparation: code for remapping to semseg format, and for relabeling masks in place
    • relabeled_data: MSeg data, annotated by Mechanical Turk workers, and verified by co-authors
    • taxonomy: on-the-fly mapping to a unified taxonomy during training, and linear mapping to evaluation taxonomies
    • utils: library functions for mask and image manipulation, filesystem, tsv/csv reading, and multiprocessing
  • tests: unit tests on all code

Data License

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.

Frequently Asked Questions (FAQ)

Q: Do the weights include the model structure or it's just the weights? If the latter, which model do these weights refer to? Under the models directory, there are several model implementations.

A: The pre-trained models follow the HRNet-W48 architecture. The model structure is defined in the code here. The saved weights provide a dictionary between keys (unique IDs for each weight identifying the corresponding layer/layer type) and values (the floating point weights).

Q: How is testing performed on the test datasets? In the paper you talk about "zero-shot transfer" -- how this is performed? Are the test dataset labels also mapped or included in the unified taxonomy? If you remapped the test dataset labels to the unified taxonomy, are the reported results the performances on the unified label space, or on each test dataset's original label space? How did you you obtain results on the WildDash dataset - which is evaluated by the server - when the MSeg taxonomy may be different from the WildDash dataset.

A: Regarding "zero-shot transfer", please refer to section "Using the MSeg taxonomy on a held-out dataset" on page 6 of our paper. This section describes how we hand-specify mappings from the unified taxonomy to each test dataset's taxonomy as a linear mapping (implemented here in mseg-api). All results are in the test dataset's original label space (i.e. if WildDash expects class indices in the range [0,18] per our names_list, our testing script uses the TaxonomyConverter transform_predictions_test() functionality to produce indices in that range, remapping probabilities.

Q: Why don't indices in MSeg_master.tsv match the training indices in individual datasets? For example, for the road class: In idd-39, road has index 0, but in idd-39-relabeled, road has index 19. It is index 7 in cityscapes-34. The cityscapes-19-relabeled index road is 11. As far as I can tell, ultimately the 'MSeg_Master.tsv' file provides the final mapping to the MSeg label space. But here, the road class seems to have an index of 98, which is neither 19 nor 11.

A: Indeed, unified taxonomy class index 98 represents "road". But we use the TaxonomyConverter to accomplish the mapping on the fly from idd-39-relabeled to the unified/universal taxonomy (we use the terms "unified" and "universal" interchangeably). This is done by adding a transform in the training loop that calls TaxonomyConverter.transform_label() on the fly. You can see how that transform is implemented here in mseg-semantic.

Q: When testing, but there are test classes that are not in the unified taxonomy (e.g. Parking, railtrack, bridge etc. in WildDash), how do you produce predictions for that class? I understand you map the predictions with a binary matrix. But what do you do when there's no one-to-one correspondence?

A: WildDash v1 uses the 19-class taxonomy for evaluation, just like Cityscapes. So we use the following script to remap the 34-class taxonomy to 19-class taxonomy for WildDash for testing inference and submission. You can see how Cityscapes evaluates just 19 of the 34 classes here in the evaluation script and in the taxonomy definition. However, bridge and rail track are actually included in our unified taxonomy, as you’ll see in MSeg_master.tsv.

Q: How are datasets images read in for training/inference? Should I use the dataset_apis from mseg-api?

A: The dataset_apis from mseg-api are not for training or inference. They are purely for generating the MSeg dataset labels on disk. We read in the datasets using mseg_semantic/utils/dataset.py and then remap them to the universal space on the fly.

Comments
  • Error in dataset path verification

    Error in dataset path verification

    When I try to run the verification script

    python -u ../tests/verify_all_dataset_paths_exist.py
    

    I get the following error:

    Finding matches for cityscapes-19-relabeled...
    Finding matches for ade20k-150-relabeled...
    Finding matches for bdd-relabeled...
    Finding matches for coco-panoptic-133-relabeled...
    Finding matches for idd-39-relabeled...
    Finding matches for sunrgbd-37-relabeled...
    Finding matches for mapillary-public65-relabeled...
    Writing visual sanity checks for ade20k-151-inst...
    Writing visual sanity checks for ade20k-151...
    On 0 of ade20k-151
    Traceback (most recent call last):
      File "../tests/verify_all_dataset_paths_exist.py", line 250, in <module>
        visual_sanitychecks()
      File "../tests/verify_all_dataset_paths_exist.py", line 94, in visual_sanitychecks
        id_to_class_name_map=id_to_classname_map
      File "/<path>/mseg-api/mseg/utils/mask_utils_detectron2.py", line 477, in overlay_instances
        class_mode_idx = get_most_populous_class(segment_mask, label_map)
      File "/<path>/mseg-api/mseg/utils/mask_utils.py", line 992, in get_most_populous_class
        class_mode_idx = get_np_mode(class_indices)
      File "/<path>/mseg-api/mseg/utils/mask_utils.py", line 931, in get_np_mode
        return np.argmax(counts)
      File "<__array_function__ internals>", line 6, in argmax
      File "/<path>/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 1186, in argmax
        return _wrapfunc(a, 'argmax', axis=axis, out=out)
      File "/<path>/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 61, in _wrapfunc
        return bound(*args, **kwds)
    ValueError: attempt to get argmax of an empty sequence
    

    Additonally adding 'ade20k-151', 'ade20k-150', and 'ade20k-150-relabeled' in line 58 to skip them, leads to the same error thrown for the BDD dataset. Thus, the error doesn't seem to be dataset specific.

    I'm running python 3.7.6 and numpy 1.18.2.

    opened by jmsteitz 26
  • Problem with COCOPanoptic dataset

    Problem with COCOPanoptic dataset

    Hello I have some problem with remap and relabelling of COCOPanoptic dataset. Firstly I needed to download it separately due to some script problem.

    First here is error from remap log, but the script completed 100% train and validation 0%. image

    Second error in relabeling log. it also says it has completed 100% train and validation 0%. image image

    After this I tried to run the unit test, and it crashes at cocopanoptic dataset. image

    opened by jdavidd 8
  • Where is supplement mentioned in the paper

    Where is supplement mentioned in the paper

    Hello, I want to know the datasets that were not used in MSeg and the reasons for not including them. However, I can not find supplement through google. Can you give me the link of supplement?

    opened by steven5401 4
  • Can't find supplement mentioned in paper

    Can't find supplement mentioned in paper

    Hi, I'm having a hard time finding the supplement material described in the paper. In a previous issue you mentioned that it will get updated to arxiv but I can't find it there. Can you let me know where should I search for this document?

    Thanks!

    opened by XAVILLA 2
  • Reintroducing the crosswalk class back into MSeg

    Reintroducing the crosswalk class back into MSeg

    Hello, I want to re-add the crosswalk class to the MSeg dataset. I have seen that mapillary vistas dataset contains the crosswalk class, and that it is relabeled to the road class. Also, I know that other datasets (such as cityscapes) contains crosswalks but they are not labeled and are considered plain road. My question is: How would you add the crosswalk class back into the dataset? . I'm thinking of 3 options, and I would appreciate your insight:

    1. Use mseg-turk to label each dataset which does not contain crosswalks. This might be the best and most straight-forward option, but this can prove labor-intensive and costly
    2. For the datasets that do not contain crosswalks, ignore the road class. For example, ignore all possible mislabels of crosswalks in the cityscapes dataset. I expect this to lower the confidence for the road class, as there will be considerably less data.
    3. Simply add the crosswalk class and revert the relabel action applied on mapillary. I woudn't try this, because it might be problematic for the model to learn crosswalks only sometimes (in case of mapillary frames) while ignoring them alltogether in other cases(e.g. cityscapes frames) - which may also create confusion between the road class and crosswalk class.

    Thanks!

    opened by ionicaML 2
  • MSEG CamVid test set: is listed but not remapped?

    MSEG CamVid test set: is listed but not remapped?

    I noticed that for camvid the test set seems to be listed (ie the TSV file camvid-11/list/test.txt exists).

    However after downloading and remapping, the semseg11 segmentation mask is missing (eg semseg11/0001TP_008550_L.png is missing), while the original 32 class segmentation mask does exists (Labels32-RGB/0001TP_008550_L.png).

    How can I remap the test set as well?

    opened by tmensink 2
  • Repeated entries for WildDash

    Repeated entries for WildDash

    In the WildDash train file list, each entry is repeated 16 times. Example repetition: https://github.com/mseg-dataset/mseg-api/blob/master/mseg/dataset_lists/wilddash-19/list/train.txt#L1 https://github.com/mseg-dataset/mseg-api/blob/master/mseg/dataset_lists/wilddash-19/list/train.txt#L71

    Is there any particular reason for that or is that an error?

    Best, Michael

    opened by gyglim 2
  • Fixes for dataset path verification script

    Fixes for dataset path verification script

    Check if polygon is degenerate, i.e. all vertices are on a horizontal line. In that case, use ImageDraw.Draw.line instead.

    Generalize type assertions by checking for subtype of np.integer instead of using a list of some integer types.

    opened by jmsteitz 2
  • Final taxonomy

    Final taxonomy

    Hello,

    thank you for interesting work.

    I have a question regarding the final class name -> id mapping. How can one obtain such mapping using the mseg-api? Are the label IDs sorted alphabetiaclly according to the universal column?

    opened by orsic 1
  • Really Nice Work!

    Really Nice Work!

    Really nice work and I hope you could release the remained 3 repos so that the community could conduct research along this direction.

    This repo is the first of 4 repos that introduce our work. It provides utilities to download the MSeg dataset (which is nontrivial), and prepare the data on disk in a unified taxonomy.

    In a few weeks, we will add the TaxonomyConverter class to this repo that supports on-the-fly mapping to a unified taxonomy during training.

    opened by PkuRainBow 1
  • Provide access to classnames of dynamically built naive taxonomy

    Provide access to classnames of dynamically built naive taxonomy

    • Logits space of "naive-merge" model should have had 316 classes, not 320 (cityscapes-34 was erroneously included in list of train datasets)
    • Style fixes (return early instead of nesting)
    opened by johnwlambert 0
  • Invitation of making PR about dataset on MMSegmentation.

    Invitation of making PR about dataset on MMSegmentation.

    Hey, here!

    I am the member of OpenMMLab and we think this dataset is very valuable for many tasks such as segmentation, we want to make more researchers and people in industry use this dataset so would you like to make a new pr on MMSegmentation with us? That could be more excellent if you would like to do it.

    Best,

    opened by MengzhangLI 1
  • Trying to reduce the number of classes of Mseg

    Trying to reduce the number of classes of Mseg

    Hi, @johnwlambert I want to make something similar with the dataset (as mentioned by someone in the issues, who wanted to add crosswalk), but also add curb and curb cut. So for this until now i figured out, that I should modify the tsv files of dataset you mentioned above. But I dont know if this is complete, so I have two questions:

    • When you said add 'crosswalk' to universal taxonomy you refer to add a new line in MSeg_master.tsv for this class? I should do this also for, curb (curb, curb cut)? are there other files, i should change?

    • I want to keep only 64 classes, so this means I only remove them (their lines) from MSeg_master.tsv and add to final line to unlabeled, and also modify the tsv file for every dataset (ex. ade20k-151_to_ade20k-150.tsv and ade20k-151_to_ade20k-150-relabeled.tsv).

    Until now I changed MSeg_master.tsv to have my selected classes, and move the other to unlabeled, and then also changed the state in every 'name_of_dataset.tsv' to unlabeled for the deleted one. I run, remap functions and everything works ok. But when I try to relabel, on ade_20k it worked, but stopped on BDD. When I run with old bdd.tsv works, with the new tsv it catch an assert:

    File "/home/ubuntu/data/datasets/mseg/mseg_files/mseg-api/mseg/utils/mask_utils.py", line 920, in swap_px_inside_mask assert np.allclose(np.unique(label_img[y,x]), np.array([old_val], dtype=np.uint8))

    This is the new bdd.tsv: bdd bdd-relabeled building building road road sidewalk sidewalk_pavement terrain terrain person unlabeled rider rider_other traffic sign traffic_sign traffic light traffic_light sky sky pole unlabeled fence unlabeled vegetation vegetation bicycle bicycle car car motorcycle motorcycle bus bus train unlabeled truck unlabeled wall wall unlabeled unlabeled

    Waiting for you response.

    opened by jdavidd 0
  • BDD dataset info

    BDD dataset info

    Downloading instruction is stating: "This dataset is not available via wget, so we ask that you download it in a browser, and upload it (1.3 GB) to the desired location on your server.

    Log in, click the "Download" tab on the left, accept terms, then click the link for "Segmentation" (file should appear under name "bdd100k_seg.zip")"

    When I try to download the file size is ~300Mb and also its name is bdd100k_sem_seg_labels_trainval. There is no file with 1.3G size and the name bdd100k_seg.zip on BDD official page (https://bdd-data.berkeley.edu/login.html)

    opened by miangoleh 7
  • Can't download ADE20K_2016_07_26.zip

    Can't download ADE20K_2016_07_26.zip

    Hi, thanks very much for your nice work. currently, I just follow your instructions and it reports that : Archive: ADE20K_2016_07_26.zip End-of-central-directory signature not found. Either this file is not a zipfile, or it constitutes one disk of a multi-part archive. In the latter case the central directory and zipfile comment will be found on the last disk(s) of this archive.

    opened by wanglixilinx 3
  • RVC 2020 Branch

    RVC 2020 Branch

    This branch provides support for our submissions to the the 2020 Robust Vision Challenge (RVC).

    It includes:

    • mappings from our universal taxonomy backwards to ADE20K, Cityscapes, KITTI, Mapillary Vistas, ScanNet, VIPER, WildDashV2. Please keep in mind that some of these backwards mappings are lossy, for example if a merge previously occurred.
    • paired test set datalists to use the mseg/label_preparation/remap_dataset.py script to prepare submissions on disk.
    • A script to analyze the per-class frequencies of any dataset, using tests/class_density_analysis.py
    • A script to visualize individual masks of all classes from any dataset, using tests/visualize_specific_class_examples.py
    opened by johnwlambert 0
Owner
null
The official repo of the CVPR 2021 paper Group Collaborative Learning for Co-Salient Object Detection .

GCoNet The official repo of the CVPR 2021 paper Group Collaborative Learning for Co-Salient Object Detection . Trained model Download final_gconet.pth

Qi Fan 46 Nov 17, 2022
Code repo for realtime multi-person pose estimation in CVPR'17 (Oral)

Realtime Multi-Person Pose Estimation By Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh. Introduction Code repo for winning 2016 MSCOCO Keypoints Cha

Zhe Cao 4.9k Dec 31, 2022
git git《Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking》(CVPR 2021) GitHub:git2] 《Masksembles for Uncertainty Estimation》(CVPR 2021) GitHub:git3]

Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking Ning Wang, Wengang Zhou, Jie Wang, and Houqiang Li Accepted by CVPR

NingWang 236 Dec 22, 2022
[CVPR 21] Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.

Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting, CVPR 2021. Ayan Kumar Bhunia, Pinaki nath Chowdhury, Yongxin Yan

Ayan Kumar Bhunia 44 Dec 12, 2022
[CVPR 2022] CoTTA Code for our CVPR 2022 paper Continual Test-Time Domain Adaptation

CoTTA Code for our CVPR 2022 paper Continual Test-Time Domain Adaptation Prerequisite Please create and activate the following conda envrionment. To r

Qin Wang 87 Jan 8, 2023
Official Repo for Ground-aware Monocular 3D Object Detection for Autonomous Driving

Visual 3D Detection Package: This repo aims to provide flexible and reproducible visual 3D detection on KITTI dataset. We expect scripts starting from

Yuxuan Liu 305 Dec 19, 2022
This repo provides the official code for TransBTS: Multimodal Brain Tumor Segmentation Using Transformer (https://arxiv.org/pdf/2103.04430.pdf).

TransBTS: Multimodal Brain Tumor Segmentation Using Transformer This repo is the official implementation for TransBTS: Multimodal Brain Tumor Segmenta

Raymond 247 Dec 28, 2022
Official repo for QHack—the quantum machine learning hackathon

Note: This repository has been frozen while we consider the submissions for the QHack Open Hackathon. We hope you enjoyed the event! Welcome to QHack,

Xanadu 118 Jan 5, 2023
The official repo of the CVPR2021 oral paper: Representative Batch Normalization with Feature Calibration

Representative Batch Normalization (RBN) with Feature Calibration The official implementation of the CVPR2021 oral paper: Representative Batch Normali

Open source projects of ShangHua-Gao 76 Nov 9, 2022
Official repo for the work titled "SharinGAN: Combining Synthetic and Real Data for Unsupervised GeometryEstimation"

SharinGAN Official repo for the work titled "SharinGAN: Combining Synthetic and Real Data for Unsupervised GeometryEstimation" The official project we

Koutilya PNVR 23 Oct 19, 2022
This Repo is the official CUDA implementation of ICCV 2019 Oral paper for CARAFE: Content-Aware ReAssembly of FEatures

Introduction This Repo is the official CUDA implementation of ICCV 2019 Oral paper for CARAFE: Content-Aware ReAssembly of FEatures. @inproceedings{Wa

Jiaqi Wang 42 Jan 7, 2023
This repo contains the official code and pre-trained models for the Dynamic Vision Transformer (DVT).

Dynamic-Vision-Transformer (Pytorch) This repo contains the official code and pre-trained models for the Dynamic Vision Transformer (DVT). Not All Ima

null 210 Dec 18, 2022
This repo contains the official implementations of EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis

EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis This repo contains the official implementations of EigenDamage: Structured Prunin

Chaoqi Wang 107 Apr 20, 2022
Official repo for SemanticGAN https://nv-tlabs.github.io/semanticGAN/

SemanticGAN This is the official code for: Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalizat

null 151 Dec 28, 2022
The official repo for CVPR2021——ViPNAS: Efficient Video Pose Estimation via Neural Architecture Search.

ViPNAS: Efficient Video Pose Estimation via Neural Architecture Search [paper] Introduction This is the official implementation of ViPNAS: Efficient V

Lumin 42 Sep 26, 2022
Official git repo for the CHIRP project

CHIRP Project This is the official git repository for the CHIRP project. Pull requests are accepted here, but for the moment, the main repository is s

Dan Smith 77 Jan 8, 2023
This repo is official PyTorch implementation of MobileHumanPose: Toward real-time 3D human pose estimation in mobile devices(CVPRW 2021).

Github Code of "MobileHumanPose: Toward real-time 3D human pose estimation in mobile devices" Introduction This repo is official PyTorch implementatio

Choi Sang Bum 203 Jan 5, 2023
Official Repo for ICCV2021 Paper: Learning to Regress Bodies from Images using Differentiable Semantic Rendering

[ICCV2021] Learning to Regress Bodies from Images using Differentiable Semantic Rendering Getting Started DSR has been implemented and tested on Ubunt

Sai Kumar Dwivedi 83 Nov 27, 2022