Code for Referring Image Segmentation via Cross-Modal Progressive Comprehension, CVPR2020.

Overview

CMPC-Refseg

Code of our CVPR 2020 paper Referring Image Segmentation via Cross-Modal Progressive Comprehension.

Shaofei Huang*, Tianrui Hui*, Si Liu, Guanbin Li, Yunchao Wei, Jizhong Han, Luoqi Liu, Bo Li (* Equal contribution)

Interpretation of CMPC.

  • (a) Input referring expression and image.

  • (b) The model first perceives all the entities described in the expression based on entity words and attribute words, e.g., “man” and “white frisbee” (orange masks and blue outline).

  • (c) After finding out all the candidate entities that may match with input expression, relational word “holding” can be further exploited to highlight the entity involved with the relationship (green arrow) and suppress the others which are not involved.

  • (d) Benefiting from the relation-aware reasoning process, the referred entity is found as the final prediction (purple mask). interpretation

Experimental Results

We modify the way of feature concatenation in the end of CMPC module and achieve higher performances than the results reported in our paper. New experimental results are summarized in the table bellow. You can download our trained checkpoints to test on the four datasets. The link to the checkpoints is: Baidu Drive, pswd: jjsf.

Method UNC val UNC testA UNC testB UNC+ val UNC+ testA UNC+ testB G-Ref val ReferIt test
STEP-ICCV19 [1] 60.04 63.46 57.97 48.19 52.33 40.41 46.40 64.13
Ours-CVPR20 61.36 64.53 59.64 49.56 53.44 43.23 49.05 65.53
Ours-Updated 62.47 65.08 60.82 50.25 54.04 43.47 49.89 65.58

Setup

We recommended the following dependencies.

  • Python 2.7
  • TensorFlow 1.5
  • Numpy
  • pydensecrf

This code is derived from RRN [2]. Please refer to it for more details of setup.

Data Preparation

  • Dataset Preprocessing

We conduct experiments on 4 datasets of referring image segmentation, including UNC, UNC+, Gref and ReferIt. After downloading these datasets, you can run the following commands for data preparation:

python build_batches.py -d Gref -t train
python build_batches.py -d Gref -t val
python build_batches.py -d unc -t train
python build_batches.py -d unc -t val
python build_batches.py -d unc -t testA
python build_batches.py -d unc -t testB
python build_batches.py -d unc+ -t train
python build_batches.py -d unc+ -t val
python build_batches.py -d unc+ -t testA
python build_batches.py -d unc+ -t testB
python build_batches.py -d referit -t trainval
python build_batches.py -d referit -t test
  • Glove Embedding

Download Gref_emb.npy and referit_emb.npy and put them in data/. We provide download link for Glove Embedding here: Baidu Drive, password: 2m28.

Training

Train on UNC training set with:

python -u trainval_model.py -m train -d unc -t train -n CMPC_model -emb -f ckpts/unc/cmpc_model

Testing

Test on UNC validation set with:

python -u trainval_model.py -m test -d unc -t val -n CMPC_model -i 700000 -c -emb -f ckpts/unc/cmpc_model

CMPC for video referring segmentation

We release video version code for CMPC on A2D dataset under CMPC_video/.

Reference

[1] Chen, Ding-Jie, et al. "See-through-text grouping for referring image segmentation." Proceedings of the IEEE International Conference on Computer Vision. 2019.

[2] Li, Ruiyu, et al. "Referring image segmentation via recurrent refinement networks." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.

Citation

If our CMPC is useful to your research, please consider citing:

@inproceedings{huang2020referring,
  title={Referring Image Segmentation via Cross-Modal Progressive Comprehension},
  author={Huang, Shaofei and Hui, Tianrui and Liu, Si and Li, Guanbin and Wei, Yunchao and Han, Jizhong and Liu, Luoqi and Li, Bo},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={10488--10497},
  year={2020}
}
You might also like...
This is an implementation for the CVPR2020 paper
This is an implementation for the CVPR2020 paper "Learning Invariant Representation for Unsupervised Image Restoration"

Learning Invariant Representation for Unsupervised Image Restoration (CVPR 2020) Introduction This is an implementation for the paper "Learning Invari

An unofficial implementation of "Unpaired Image Super-Resolution using Pseudo-Supervision." CVPR2020

UnpairedSR An unofficial implementation of "Unpaired Image Super-Resolution using Pseudo-Supervision." CVPR2020 turn RCAN(modified) -- xmodel(xilinx

RCDNet: A Model-driven Deep Neural Network for Single Image Rain Removal (CVPR2020)

RCDNet: A Model-driven Deep Neural Network for Single Image Rain Removal (CVPR2020) Hong Wang, Qi Xie, Qian Zhao, and Deyu Meng [PDF] [Supplementary M

A pytorch-based deep learning framework for multi-modal 2D/3D medical image segmentation
A pytorch-based deep learning framework for multi-modal 2D/3D medical image segmentation

A 3D multi-modal medical image segmentation library in PyTorch We strongly believe in open and reproducible deep learning research. Our goal is to imp

Code for the Active Speakers in Context Paper (CVPR2020)

Active Speakers in Context This repo contains the official code and models for the "Active Speakers in Context" CVPR 2020 paper. Before Training The c

《Image2Reverb: Cross-Modal Reverb Impulse Response Synthesis》(2021)

Image2Reverb Image2Reverb is an end-to-end neural network that generates plausible audio impulse responses from single images of acoustic environments

 Cross-modal Deep Face Normals with Deactivable Skip Connections
Cross-modal Deep Face Normals with Deactivable Skip Connections

Cross-modal Deep Face Normals with Deactivable Skip Connections Victoria Fernández Abrevaya*, Adnane Boukhayma*, Philip H. S. Torr, Edmond Boyer (*Equ

 Probabilistic Cross-Modal Embedding (PCME) CVPR 2021
Probabilistic Cross-Modal Embedding (PCME) CVPR 2021

Probabilistic Cross-Modal Embedding (PCME) CVPR 2021 Official Pytorch implementation of PCME | Paper Sanghyuk Chun1 Seong Joon Oh1 Rafael Sampaio de R

Official Implement of CVPR 2021 paper “Cross-Modal Collaborative Representation Learning and a Large-Scale RGBT Benchmark for Crowd Counting”
Official Implement of CVPR 2021 paper “Cross-Modal Collaborative Representation Learning and a Large-Scale RGBT Benchmark for Crowd Counting”

RGBT Crowd Counting Lingbo Liu, Jiaqi Chen, Hefeng Wu, Guanbin Li, Chenglong Li, Liang Lin. "Cross-Modal Collaborative Representation Learning and a L

Comments
  • Not found: Key fc1_voc12_c0/biases not found in checkpoint

    Not found: Key fc1_voc12_c0/biases not found in checkpoint

    Thank you for sharing your project, including the pretrained model weights. It works well when I train the network from the dataset and test it. But, when I tried to load the pretrained weights, an error occurs. Could you please help me? Not found: Key fc1_voc12_c0/biases not found in checkpoint Traceback (most recent call last): File "/home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call return fn(*args) File "/home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn target_list, run_metadata) File "/home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.NotFoundError: 2 root error(s) found. (0) Not found: Key fc1_voc12_c0/biases not found in checkpoint [[{{node save/RestoreV2}}]] (1) Not found: Key fc1_voc12_c0/biases not found in checkpoint [[{{node save/RestoreV2}}]] [[save/RestoreV2/_301]] 0 successful operations. 0 derived errors ignored.

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last): File "/home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py", line 1290, in restore {self.saver_def.filename_tensor_name: save_path}) File "/home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 956, in run run_metadata_ptr) File "/home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run feed_dict_tensor, options, run_metadata) File "/home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run run_metadata) File "/home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.NotFoundError: 2 root error(s) found. (0) Not found: Key fc1_voc12_c0/biases not found in checkpoint [[node save/RestoreV2 (defined at /home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) ]] (1) Not found: Key fc1_voc12_c0/biases not found in checkpoint [[node save/RestoreV2 (defined at /home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) ]] [[save/RestoreV2/_301]] 0 successful operations. 0 derived errors ignored.

    Original stack trace for 'save/RestoreV2': File "trainval_model.py", line 354, in pre_emb=args.emb) File "trainval_model.py", line 172, in test snapshot_restorer = tf.train.Saver() File "/home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py", line 828, in init self.build() File "/home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py", line 840, in build self._build(self._filename, build_save=True, build_restore=True) File "/home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py", line 878, in _build build_restore=build_restore) File "/home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py", line 508, in _build_internal restore_sequentially, reshape) File "/home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py", line 328, in _AddRestoreOps restore_sequentially) File "/home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py", line 575, in bulk_restore return io_ops.restore_v2(filename_tensor, names, slices, dtypes) File "/home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/ops/gen_io_ops.py", line 1696, in restore_v2 name=name) File "/home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper op_def=op_def) File "/home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/util/deprecation.py", line 513, in new_func return func(*args, **kwargs) File "/home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op attrs, op_def, compute_device) File "/home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal op_def=op_def) File "/home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in init self._traceback = tf_stack.extract_stack()

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last): File "/home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py", line 1300, in restore names_to_keys = object_graph_key_mapping(save_path) File "/home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py", line 1618, in object_graph_key_mapping object_graph_string = reader.get_tensor(trackable.OBJECT_GRAPH_PROTO_KEY) File "/home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/pywrap_tensorflow_internal.py", line 915, in get_tensor return CheckpointReader_GetTensor(self, compat.as_bytes(tensor_str)) tensorflow.python.framework.errors_impl.NotFoundError: Key _CHECKPOINTABLE_OBJECT_GRAPH not found in checkpoint

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last): File "trainval_model.py", line 354, in pre_emb=args.emb) File "trainval_model.py", line 178, in test snapshot_restorer.restore(sess, weights) File "/home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py", line 1306, in restore err, "a Variable name or other graph key that is missing") tensorflow.python.framework.errors_impl.NotFoundError: Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

    2 root error(s) found. (0) Not found: Key fc1_voc12_c0/biases not found in checkpoint [[node save/RestoreV2 (defined at /home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) ]] (1) Not found: Key fc1_voc12_c0/biases not found in checkpoint [[node save/RestoreV2 (defined at /home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) ]] [[save/RestoreV2/_301]] 0 successful operations. 0 derived errors ignored.

    Original stack trace for 'save/RestoreV2': File "trainval_model.py", line 354, in pre_emb=args.emb) File "trainval_model.py", line 172, in test snapshot_restorer = tf.train.Saver() File "/home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py", line 828, in init self.build() File "/home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py", line 840, in build self._build(self._filename, build_save=True, build_restore=True) File "/home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py", line 878, in _build build_restore=build_restore) File "/home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py", line 508, in _build_internal restore_sequentially, reshape) File "/home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py", line 328, in _AddRestoreOps restore_sequentially) File "/home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py", line 575, in bulk_restore return io_ops.restore_v2(filename_tensor, names, slices, dtypes) File "/home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/ops/gen_io_ops.py", line 1696, in restore_v2 name=name) File "/home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper op_def=op_def) File "/home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/util/deprecation.py", line 513, in new_func return func(*args, **kwargs) File "/home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op attrs, op_def, compute_device) File "/home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal op_def=op_def) File "/home/XXXXXX/Programs/anaconda3/envs/cmpc-env/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in init self._traceback = tf_stack.extract_stack()

    opened by zhuyamanashi2016 0
  • referit checkpoint file corrupted

    referit checkpoint file corrupted

    Hi, I saw that referit checkpoint file is ~323 MB, while the other dataset's checpoints are ~1.01 GB. When I evaluate using referit checkpoint, I get out of range error. Can you please share the full trained checkpoint for referit? Thanks

    opened by kanji95 1
  • environment error

    environment error

    environment setup: tensorflow-gpu=1.5 & cudnn=7.6 raise core dump error:Loaded runtime CuDNN library: 7605 (compatibility version 7600) but source was compiled with 7102 (compatibility version 7100).

    opened by SxJyJay 3
Owner
spyflying
Two students of Cola Lab, BUAA.
spyflying
PyTorch code for the paper "Complementarity is the King: Multi-modal and Multi-grained Hierarchical Semantic Enhancement Network for Cross-modal Retrieval".

Complementarity is the King: Multi-modal and Multi-grained Hierarchical Semantic Enhancement Network for Cross-modal Retrieval (M2HSE) PyTorch code fo

Xinlei-Pei 6 Dec 23, 2022
Vision-Language Transformer and Query Generation for Referring Segmentation (ICCV 2021)

Vision-Language Transformer and Query Generation for Referring Segmentation Please consider citing our paper in your publications if the project helps

Henghui Ding 143 Dec 23, 2022
Code for 'Single Image 3D Shape Retrieval via Cross-Modal Instance and Category Contrastive Learning', ICCV 2021

CMIC-Retrieval Code for Single Image 3D Shape Retrieval via Cross-Modal Instance and Category Contrastive Learning. ICCV 2021. Introduction In this wo

null 42 Nov 17, 2022
CVPR 2021 Official Pytorch Code for UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training

UC2 UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training Mingyang Zhou, Luowei Zhou, Shuohang Wang, Yu Cheng, Linjie Li, Zhou Yu,

Mingyang Zhou 28 Dec 30, 2022
ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integration

ROSITA News & Updates (24/08/2021) Release the demo to perform fine-grained semantic alignments using the pretrained ROSITA model. (15/08/2021) Releas

Vision and Language Group@ MIL 48 Dec 23, 2022
Code of U2Fusion: a unified unsupervised image fusion network for multiple image fusion tasks, including multi-modal, multi-exposure and multi-focus image fusion.

U2Fusion Code of U2Fusion: a unified unsupervised image fusion network for multiple image fusion tasks, including multi-modal (VIS-IR, medical), multi

Han Xu 129 Dec 11, 2022
Cross-Modal Contrastive Learning for Text-to-Image Generation

Cross-Modal Contrastive Learning for Text-to-Image Generation This repository hosts the open source JAX implementation of XMC-GAN. Setup instructions

Google Research 94 Nov 12, 2022
Implementation for the EMNLP 2021 paper "Interactive Machine Comprehension with Dynamic Knowledge Graphs".

Interactive Machine Comprehension with Dynamic Knowledge Graphs Implementation for the EMNLP 2021 paper. Dependencies apt-get -y update apt-get instal

Xingdi (Eric) Yuan 19 Aug 23, 2022
Datasets accompanying the paper ConditionalQA: A Complex Reading Comprehension Dataset with Conditional Answers.

ConditionalQA Datasets accompanying the paper ConditionalQA: A Complex Reading Comprehension Dataset with Conditional Answers. Disclaimer This dataset

null 2 Oct 14, 2021
Pytorch code for ICRA'21 paper: "Hierarchical Cross-Modal Agent for Robotics Vision-and-Language Navigation"

Hierarchical Cross-Modal Agent for Robotics Vision-and-Language Navigation This repository is the pytorch implementation of our paper: Hierarchical Cr

null 43 Nov 21, 2022