Transparent Transformer Segmentation

谢恩泽

Last update: Jan 2, 2023

Related tags

Deep Learning Trans2Seg

Overview

Transparent Transformer Segmentation

Introduction

This repository contains the data and code for IJCAI 2021 paper Segmenting transparent object in the wild with transformer.

Environments

python 3
torch = 1.4.0
torchvision
pyyaml
Pillow
numpy

INSTALL

python setup.py develop --user

Data Preparation

create dirs './datasets/transparent/Trans10K_v2'
put the train/validation/test data under './datasets/transparent/Trans10K_v2'. Data Structure is shown below.

Trans10K_v2
├── test
│   ├── images
│   └── masks_12
├── train
│   ├── images
│   └── masks_12
└── validation
    ├── images
    └── masks_12

Download Dataset: Google Drive. Baidu Drive. code: oqms

Network Define

The code of Network pipeline is in segmentron/models/trans2seg.py.

The code of Transformer Encoder-Decoder is in segmentron/modules/transformer.py.

Train

Our experiments are based on one machine with 8 V100 GPUs with 32g memory, about 1 hour training time.

bash tools/dist_train.sh $CONFIG-FILE $GPUS

For example:

bash tools/dist_train.sh configs/trans10kv2/trans2seg/trans2seg_medium.yaml 8

Test

bash tools/dist_train.sh $CONFIG-FILE $GPUS --test TEST.TEST_MODEL_PATH $MODEL_PATH

Citations

Please consider citing our paper in your publications if the project helps your research. BibTeX reference is as follows.

@article{xie2021segmenting,
  title={Segmenting transparent object in the wild with transformer},
  author={Xie, Enze and Wang, Wenjia and Wang, Wenhai and Sun, Peize and Xu, Hang and Liang, Ding and Luo, Ping},
  journal={arXiv preprint arXiv:2101.08461},
  year={2021}
}

Comments

Cannot follow the train demo

When I followed the train demo, there were some path errors like

(trans2seg) jinrae@costar-Oryx-Pro:~/workspace/Trans2Seg$ bash tools/dist_train.sh configs/trans10kv2/trans2seg/trans2seg_medium.yaml 1
+ CONFIG=configs/trans10kv2/trans2seg/trans2seg_medium.yaml
+ GPUS=8
++ dirname tools/dist_train.sh
+ python -m torch.distributed.launch --nproc_per_node=8 tools/train.py --config-file configs/trans10kv2/trans2seg/trans2seg_medium.yaml 1
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
*****************************************
Traceback (most recent call last):
  File "tools/train.py", line 20, in <module>
    from segmentron.solver.loss import get_segmentation_loss
  File "/home/jinrae/workspace/Trans2Seg/segmentron/solver/loss.py", line 9, in <module>
Traceback (most recent call last):
  File "tools/train.py", line 20, in <module>
    from ..models.pointrend import point_sample
ModuleNotFoundError: No module named 'segmentron.models.pointrend'
    from segmentron.solver.loss import get_segmentation_loss
  File "/home/jinrae/workspace/Trans2Seg/segmentron/solver/loss.py", line 9, in <module>
    from ..models.pointrend import point_sample
ModuleNotFoundError: No module named 'segmentron.models.pointrend'
Traceback (most recent call last):
  File "tools/train.py", line 20, in <module>
    from segmentron.solver.loss import get_segmentation_loss
  File "/home/jinrae/workspace/Trans2Seg/segmentron/solver/loss.py", line 9, in <module>
    from ..models.pointrend import point_sample
ModuleNotFoundError: No module named 'segmentron.models.pointrend'
Traceback (most recent call last):
  File "tools/train.py", line 20, in <module>
    from segmentron.solver.loss import get_segmentation_loss
  File "/home/jinrae/workspace/Trans2Seg/segmentron/solver/loss.py", line 9, in <module>
    from ..models.pointrend import point_sample
ModuleNotFoundError: No module named 'segmentron.models.pointrend'
Traceback (most recent call last):
  File "/home/jinrae/miniconda3/envs/trans2seg/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/jinrae/miniconda3/envs/trans2seg/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/jinrae/miniconda3/envs/trans2seg/lib/python3.6/site-packages/torch/distributed/launch.py", line 263, in <module>
    main()
  File "/home/jinrae/miniconda3/envs/trans2seg/lib/python3.6/site-packages/torch/distributed/launch.py", line 259, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/home/jinrae/miniconda3/envs/trans2seg/bin/python', '-u', 'tools/train.py', '--local_rank=7', '--config-file', 'configs/trans10kv2/trans2seg/trans2seg_medium.yaml', '1']' returned non-zero exit status 1.
Traceback (most recent call last):
  File "tools/train.py", line 20, in <module>
    from segmentron.solver.loss import get_segmentation_loss
  File "/home/jinrae/workspace/Trans2Seg/segmentron/solver/loss.py", line 9, in <module>
    from ..models.pointrend import point_sample
ModuleNotFoundError: No module named 'segmentron.models.pointrend'
Traceback (most recent call last):
  File "tools/train.py", line 20, in <module>
    from segmentron.solver.loss import get_segmentation_loss
  File "/home/jinrae/workspace/Trans2Seg/segmentron/solver/loss.py", line 9, in <module>
    from ..models.pointrend import point_sample
ModuleNotFoundError: No module named 'segmentron.models.pointrend'
Traceback (most recent call last):
  File "tools/train.py", line 20, in <module>
    from segmentron.solver.loss import get_segmentation_loss
  File "/home/jinrae/workspace/Trans2Seg/segmentron/solver/loss.py", line 9, in <module>
    from ..models.pointrend import point_sample
ModuleNotFoundError: No module named 'segmentron.models.pointrend'
(trans2seg) jinrae@costar-Oryx-Pro:~/workspace/Trans2Seg$ Traceback (most recent call last):
  File "tools/train.py", line 20, in <module>
    from segmentron.solver.loss import get_segmentation_loss
  File "/home/jinrae/workspace/Trans2Seg/segmentron/solver/loss.py", line 9, in <module>
    from ..models.pointrend import point_sample
ModuleNotFoundError: No module named 'segmentron.models.pointrend'

opened by JinraeKim 1

How and where should I modify the code when I am reducing the class

When I try to reduce the categories to two while testing, I get the following error：

THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1579027003190/work/aten/src/THC/generic/THCTensorMath.cu line=26 error=59 : device-side assert triggered Traceback (most recent call last): File "tools/train.py", line 330, in trainer.train() File "tools/train.py", line 170, in train losses.backward() File "/opt/conda/envs/trans_seg/lib/python3.6/site-packages/torch/tensor.py", line 195, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph) File "/opt/conda/envs/trans_seg/lib/python3.6/site-packages/torch/autograd/init.py", line 99, in backward allow_unreachable=True) # allow_unreachable flag RuntimeError: cuda runtime error (59) : device-side assert triggered at /opt/conda/conda-bld/pytorch_1579027003190/work/aten/src/THC/generic/THCTensorMath.cu:26 Traceback (most recent call last): File "/opt/conda/envs/trans_seg/lib/python3.6/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/opt/conda/envs/trans_seg/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/opt/conda/envs/trans_seg/lib/python3.6/site-packages/torch/distributed/launch.py", line 263, in main() File "/opt/conda/envs/trans_seg/lib/python3.6/site-packages/torch/distributed/launch.py", line 259, in main cmd=cmd)

How to modify the network parameters to run？ thank you！

opened by xhangHU 1
CVE-2007-4559 Patch

Patching CVE-2007-4559

Hi, we are security researchers from the Advanced Research Center at Trellix. We have began a campaign to patch a widespread bug named CVE-2007-4559. CVE-2007-4559 is a 15 year old bug in the Python tarfile package. By using extract() or extractall() on a tarfile object without sanitizing input, a maliciously crafted .tar file could perform a directory path traversal attack. We found at least one unsantized extractall() in your codebase and are providing a patch for you via pull request. The patch essentially checks to see if all tarfile members will be extracted safely and throws an exception otherwise. We encourage you to use this patch or your own solution to secure against CVE-2007-4559. Further technical information about the vulnerability can be found in this blog.

If you have further questions you may contact us through this projects lead researcher Kasimir Schulz.

opened by TrellixVulnTeam 0
about demo

Does anyone knows how to use the demo?It always has a AssertionError when i use it. File "E:\python\Trans2Seg-master\segmentron\models\trans2seg.py", line 74, in forward cls_token, x = self.vit.forward_encoder(x) File "E:\python\Trans2Seg-master\segmentron\modules\transformer.py", line 224, in forward_encoder pos_embed = self.resize_pos_embed(x, pos_embed) File "E:\python\Trans2Seg-master\segmentron\modules\transformer.py", line 242, in resize_pos_embed assert x_h * x_w == hw-1 AssertionError

opened by WTsPython 5
ModuleNotFoundError: No module named 'segmentron.models.pointrend'
Hi,

Have been trying to run training (though on single GPU). -> bash tools/dist_train.sh configs/trans10kv2/trans2seg/trans2seg_medium.yaml 1

And below is error stack. Please can you help? Many thanks.

CONFIG=configs/trans10kv2/trans2seg/trans2seg_medium.yaml

GPUS=8 ++ dirname tools/dist_train.sh

python -m torch.distributed.launch --nproc_per_node=8 tools/train.py --config-file configs/trans10kv2/trans2seg/trans2seg_medium.yaml 1

Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.

Traceback (most recent call last): File "tools/train.py", line 20, in from segmentron.solver.loss import get_segmentation_loss File "/home/paperspace/Trans2Seg/segmentron/solver/loss.py", line 9, in from ..models.pointrend import point_sample ModuleNotFoundError: No module named 'segmentron.models.pointrend' Traceback (most recent call last): File "tools/train.py", line 20, in from segmentron.solver.loss import get_segmentation_loss File "/home/paperspace/Trans2Seg/segmentron/solver/loss.py", line 9, in from ..models.pointrend import point_sample ModuleNotFoundError: No module named 'segmentron.models.pointrend' Traceback (most recent call last): File "tools/train.py", line 20, in from segmentron.solver.loss import get_segmentation_loss File "/home/paperspace/Trans2Seg/segmentron/solver/loss.py", line 9, in from ..models.pointrend import point_sample ModuleNotFoundError: No module named 'segmentron.models.pointrend' Traceback (most recent call last): File "tools/train.py", line 20, in from segmentron.solver.loss import get_segmentation_loss File "/home/paperspace/Trans2Seg/segmentron/solver/loss.py", line 9, in from ..models.pointrend import point_sample ModuleNotFoundError: No module named 'segmentron.models.pointrend' Traceback (most recent call last): File "tools/train.py", line 20, in from segmentron.solver.loss import get_segmentation_loss File "/home/paperspace/Trans2Seg/segmentron/solver/loss.py", line 9, in from ..models.pointrend import point_sample ModuleNotFoundError: No module named 'segmentron.models.pointrend' Traceback (most recent call last): File "tools/train.py", line 20, in from segmentron.solver.loss import get_segmentation_loss File "/home/paperspace/Trans2Seg/segmentron/solver/loss.py", line 9, in from ..models.pointrend import point_sample ModuleNotFoundError: No module named 'segmentron.models.pointrend' Traceback (most recent call last): File "tools/train.py", line 20, in from segmentron.solver.loss import get_segmentation_loss File "/home/paperspace/Trans2Seg/segmentron/solver/loss.py", line 9, in from ..models.pointrend import point_sample ModuleNotFoundError: No module named 'segmentron.models.pointrend' Traceback (most recent call last): File "tools/train.py", line 20, in from segmentron.solver.loss import get_segmentation_loss File "/home/paperspace/Trans2Seg/segmentron/solver/loss.py", line 9, in from ..models.pointrend import point_sample ModuleNotFoundError: No module named 'segmentron.models.pointrend' Traceback (most recent call last): File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/paperspace/Torch/lib/python3.8/site-packages/torch/distributed/launch.py", line 263, in main() File "/home/paperspace/Torch/lib/python3.8/site-packages/torch/distributed/launch.py", line 258, in main raise subprocess.CalledProcessError(returncode=process.returncode, subprocess.CalledProcessError: Command '['/home/paperspace/Torch/bin/python', '-u', 'tools/train.py', '--local_rank=7', '--config-file', 'configs/trans10kv2/trans2seg/trans2seg_medium.yaml', '1']' returned non-zero exit status 1.
opened by kontramind 1

Transparent Transformer Segmentation

Related tags

Overview

Transparent Transformer Segmentation

Introduction

Environments

INSTALL

Data Preparation

Network Define

Train

Test

Citations

Comments

Cannot follow the train demo

How and where should I modify the code when I am reducing the class

CVE-2007-4559 Patch

Patching CVE-2007-4559

about demo

ModuleNotFoundError: No module named 'segmentron.models.pointrend'

Owner

谢恩泽

RGB-D Local Implicit Function for Depth Completion of Transparent Objects

VSR-Transformer - This paper proposes a new Transformer for video super-resolution (called VSR-Transformer).

This is an official implementation for "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" on Object Detection and Instance Segmentation.

Implementation of Transformer in Transformer, pixel level attention paired with patch level attention for image classification, in Pytorch

Third party Pytorch implement of Image Processing Transformer (Pre-Trained Image Processing Transformer arXiv:2012.00364v2)

Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch

Episodic Transformer (E.T.) is a novel attention-based architecture for vision-and-language navigation. E.T. is based on a multimodal transformer that encodes language inputs and the full episode history of visual observations and actions.

The implementation of "Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer"

Unofficial implementation of "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" (https://arxiv.org/abs/2103.14030)

CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped

3D-Transformer: Molecular Representation with Transformer in 3D Space

This repository builds a basic vision transformer from scratch so that one beginner can understand the theory of vision transformer.

Transformer - Transformer in PyTorch

Transformer Huffman coding - Complete Huffman coding through transformer

Implementation of the Transformer variant proposed in "Transformer Quality in Linear Time"

Pytorch Code for "Medical Transformer: Gated Axial-Attention for Medical Image Segmentation"

This repo provides the official code for TransBTS: Multimodal Brain Tumor Segmentation Using Transformer (https://arxiv.org/pdf/2103.04430.pdf).

CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation

"SOLQ: Segmenting Objects by Learning Queries", SOLQ is an end-to-end instance segmentation framework with Transformer.