Segmentation from Natural Language Expressions
This repository contains the Caffe reimplementation of the following paper:
- R. Hu, M. Rohrbach, T. Darrell, Segmentation from Natural Language Expressions. in arXiv:1603.06180, 2016. (PDF)
@article{hu2016segmentation,
title={Segmentation from Natural Language Expressions},
author={Hu, Ronghang and Rohrbach, Marcus and Darrell, Trevor},
journal={arXiv preprint arXiv:1603.06180},
year={2016}
}
Project Page: http://ronghanghu.com/text_objseg
Installation
- Install Caffe following the instructions here.
- Download this repository or clone with Git, and then
cdinto the root directory of the repository.
Training and evaluation on ReferIt Dataset
Download dataset and VGG network
Download ReferIt dataset:
./referit/referit-dataset/download_referit_dataset.sh
Download the caffemodel for VGG-16 network parameters trained on ImageNET 1000 classes.
Training
You may need to add the repository root directory to Python's module path:
export PYTHONPATH=/path/to/text_objseg_caffe/:$PYTHONPATH
Build training batches for bounding boxes:
python referit/build_training_batches_det.py
Build training batches for segmentation:
python referit/build_training_batches_seg.py
Configure the config.py file in the directory det_model and train the language-based bounding box localization model:
python det_model/train_det_model.py
Configure the config.py file in the directory seg_low_res_model and train the low resolution language-based segmentation model (from the previous bounding box localization model):
python seg_low_res_model/train_low_res_model.py
Configure the config.py file in the directory seg_model and train the high resolution language-based segmentation model (from the previous low resolution segmentation model):
python seg_model/train_seg_model.py
Evaluation
You may need to add the repository root directory to Python's module path:
export PYTHONPATH=path/to/text_objseg_caffe:$PYTHONPATH
Configure the test_config.py file in the directory seg_model and run evaluation for the high resolution language-based segmentation model:
python seg_model/test_seg_model.py
This should reproduce the results in the paper. You may also evaluate the language-based bounding box localization model:
python det_model/test_det_model.py
The results can be compared to this paper.
Demo
There is a demo that you can try! Run the demo in ./demo/text_objseg_demo.ipynb with Jupyter Notebook (IPython Notebook).
11 Nov 28, 2022
105 Nov 25, 2022
8.4k Jan 3, 2023
14 Sep 1, 2022
83 Jan 5, 2023
35 Oct 31, 2022
49 Dec 22, 2022
82 Dec 29, 2022
5.7k Jan 9, 2023
365 Dec 31, 2022
9 Nov 4, 2022
1 Feb 18, 2022
9.6k Dec 31, 2022
89 Dec 13, 2022
38 Dec 11, 2022
278 Dec 11, 2022