The "cls_token" and "pos_embed" are defined as all-zero matrices, what is the meaning? How is it applied in the model later? I am not doing this direction, just want to learn from your work, but also hope that you can help me answer!

"self.cls_token = nn.Parameter(torch.zeros(1, 1, self.embed_dim)) self.pos_embed = nn.Parameter(torch.zeros( 1, self.num_patches + 1, self.embed_dim)) "

Since the patches come from a 2D images, the position information consists of two directions, in other words, x-axis and y-axis indexes. This is different from the case in 1-D sequence. How do you implement the position embedding? Can you share the details since the code is not released?

I cannt achieve the most miou proposed in original paper while i didnt change the hyperparameter except tuning the bs from 8 to 4. How to achieve the best miou? Is that related to batchsize?

Hello，thanks for your code. How much GPU memory is needed for training SETR ? I have 2 P40 GPU but I cann't start training cus OOM. Looking forward to your reply.

Hi, do you have a google drive link for the models with T-Base referenced in the paper (such as SETR-Naive-Base) as well as the corresponding configuration files?

Alternatively, what configuration can I use to train the model if it is not readily available? I tried changing the depth in SETR/configs/base/models/setr_naive_pup.py to 12, but that errors out with "RuntimeError: shape '[2, 1025, 3, 12, 85]' is invalid for input of size 6297600" when using the ADE20K configuration file (https://github.com/fudan-zvg/SETR/blob/main/configs/SETR/SETR_Naive_512x512_160k_ade20k_bs_16.py) for training. Changing the embedding dimension in this file from 1024 results in a lot of shape mismatches with the pretrained imagenet21k model as well. The default training with the T-large depth and embedding dimension work for me with the same file.

Thanks for your help.

Can you help me solve this problem? I use the dataset in VOC format.

Traceback (most recent call last): File "tools/train.py", line 163, in main() File "tools/train.py", line 159, in main meta=meta) File "/home/ubuntu/disk1/user/SETR-main/mmseg/apis/train.py", line 91, in train_segmentor val_dataset = build_dataset(cfg.data.val, dict(test_mode=True)) File "/home/ubuntu/disk1/user/SETR-main/mmseg/datasets/builder.py", line 73, in build_dataset dataset = build_from_cfg(cfg, DATASETS, default_args) File "/home/ubuntu/anaconda3/envs/setr/lib/python3.7/site-packages/mmcv/utils/registry.py", line 171, in build_from_cfg return obj_cls(**args) File "/home/ubuntu/disk1/user/SETR-main/mmseg/datasets/pascal_context.py", line 53, in init **kwargs) File "/home/ubuntu/disk1/user/SETR-main/mmseg/datasets/custom.py", line 86, in init self.pipeline = Compose(pipeline) File "/home/ubuntu/disk1/user/SETR-main/mmseg/datasets/pipelines/compose.py", line 22, in init transform = build_from_cfg(transform, PIPELINES) File "/home/ubuntu/anaconda3/envs/setr/lib/python3.7/site-packages/mmcv/utils/registry.py", line 171, in build_from_cfg return obj_cls(**args) File "/home/ubuntu/disk1/user/SETR-main/mmseg/datasets/pipelines/test_time_aug.py", line 59, in init self.transforms = Compose(transforms) File "/home/ubuntu/disk1/user/SETR-main/mmseg/datasets/pipelines/compose.py", line 22, in init transform = build_from_cfg(transform, PIPELINES) File "/home/ubuntu/anaconda3/envs/setr/lib/python3.7/site-packages/mmcv/utils/registry.py", line 171, in build_from_cfg return obj_cls(**args) TypeError: init() got an unexpected keyword argument 'dataset'

@lzrobots @VictorLlu @sixiaozheng Hi, thank you for your sharing. however, when i run "./tools/dist_test.sh configs/SETR/SETR_PUP_512x512_160k_ade20k_bs_16_MS.py", i got the error: CUDA out of memory. I have 2 NVIDIA Tesla P100 about 16GB per GPU. Could you please tell me what is wrong. Thank you.

Hi, authors,

I got the following error after executing command: python tools/train.py configs/SETR/SETR_PUP_768x768_40k_cityscapes_bs_8.py

2021-04-08 08:03:22,265 - mmseg - INFO - Loaded 2975 images
2021-04-08 08:03:24,275 - mmseg - INFO - Loaded 500 images
2021-04-08 08:03:24,276 - mmseg - INFO - Start running, host: root@milton-LabPC, work_dir: /media/root/mdata/data/code13/SETR/work_dirs/SETR_PUP_768x768_40k_cityscapes_bs_8
2021-04-08 08:03:24,276 - mmseg - INFO - workflow: [('train', 1)], max: 40000 iters
Traceback (most recent call last):
  File "tools/train.py", line 161, in <module>
    main()
  File "tools/train.py", line 150, in main
    train_segmentor(
  File "/media/root/mdata/data/code13/SETR/mmseg/apis/train.py", line 106, in train_segmentor
    runner.run(data_loaders, cfg.workflow, cfg.total_iters)
  File "/root/anaconda3/envs/pytorch1.7.0/lib/python3.8/site-packages/mmcv/runner/iter_based_runner.py", line 130, in run
    iter_runner(iter_loaders[i], **kwargs)
  File "/root/anaconda3/envs/pytorch1.7.0/lib/python3.8/site-packages/mmcv/runner/iter_based_runner.py", line 60, in train
    outputs = self.model.train_step(data_batch, self.optimizer, **kwargs)
  File "/root/anaconda3/envs/pytorch1.7.0/lib/python3.8/site-packages/mmcv/parallel/data_parallel.py", line 67, in train_step
    return self.module.train_step(*inputs[0], **kwargs[0])
  File "/media/root/mdata/data/code13/SETR/mmseg/models/segmentors/base.py", line 152, in train_step
    losses = self(**data_batch)
  File "/root/anaconda3/envs/pytorch1.7.0/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/root/anaconda3/envs/pytorch1.7.0/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 84, in new_func
    return old_func(*args, **kwargs)
  File "/media/root/mdata/data/code13/SETR/mmseg/models/segmentors/base.py", line 122, in forward
    return self.forward_train(img, img_metas, **kwargs)
  File "/media/root/mdata/data/code13/SETR/mmseg/models/segmentors/encoder_decoder.py", line 157, in forward_train
    loss_decode = self._decode_head_forward_train(x, img_metas,
  File "/media/root/mdata/data/code13/SETR/mmseg/models/segmentors/encoder_decoder.py", line 100, in _decode_head_forward_train
    loss_decode = self.decode_head.forward_train(x, img_metas,
  File "/media/root/mdata/data/code13/SETR/mmseg/models/decode_heads/decode_head.py", line 185, in forward_train
    seg_logits = self.forward(inputs)
  File "/media/root/mdata/data/code13/SETR/mmseg/models/decode_heads/vit_up_head.py", line 93, in forward
    x = self.syncbn_fc_0(x)
  File "/root/anaconda3/envs/pytorch1.7.0/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/root/anaconda3/envs/pytorch1.7.0/lib/python3.8/site-packages/torch/nn/modules/batchnorm.py", line 519, in forward
    world_size = torch.distributed.get_world_size(process_group)
  File "/root/anaconda3/envs/pytorch1.7.0/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 625, in get_world_size
    return _get_group_size(group)
  File "/root/anaconda3/envs/pytorch1.7.0/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 220, in _get_group_size
    _check_default_pg()
  File "/root/anaconda3/envs/pytorch1.7.0/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 210, in _check_default_pg
    assert _default_pg is not None, \
AssertionError: Default process group is not initialized
(pytorch1.7.0) root@milton-LabPC:/data/code13/SETR

As I use a single GPU device to perform the training, it seems the error is related to distributed training. Any hints to solve this issue?

THX!

@lzrobots The paper seems promising, but some question about the efficiency are unanswered:

• For CPU-only inferencing, how much memory is required for inferencing a 1024 x 1024 image?
• For CPU-only inferencing, what is the fps count for 1024*1024 images?
• Number of FLOPS and Parameters?

After reading your paper, I have a confusion that how do you handle the multi-patch (256) inputs in the encoder? It seems that in the encoder, the network fuses the 256 patches and learns one feature map (with size: (H/16, W/16, D)) of the whole original image (instead of the patch-wise image), and then decode this feature map to generate the segmentatoin map. Wonder how to process and fue the 256 patches in the encoder?

Excuse me I'm Trainning with multiple GPUs,for example:./tools/dist_train.sh ${CONFIG_FILE} ${GPU_NUM} [optional arguments] and I have 2 GPUs try to use Traceback (most recent call last): File "/home/anaconda3/envs/py37_torch1.6/lib/python3.7/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/home/anaconda3/envs/py37_torch1.6/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/anaconda3/envs/py37_torch1.6/lib/python3.7/site-packages/torch/distributed/launch.py", line 261, in main() File "/home/anaconda3/envs/py37_torch1.6/lib/python3.7/site-packages/torch/distributed/launch.py", line 257, in main cmd=cmd) subprocess.CalledProcessError: Command '['/home/anaconda3/envs/py37_torch1.6/bin/python', '-u', './tools/train.py', '--local_rank=1', 'configs/SETR/SETR_Naive_768x768_40k_cityscapes_bs_8.py', '--launcher', 'pytorch', '--load-from=./pth/jx_vit_large_p16_384-b3be5167.pth']' returned non-zero exit status 1.

Thanks for your answer！

I encountered the error: **RecursionError: maximum recursion depth exceeded in comparison ** while training with my custom dataset. I tried to set num_workers to 0 but the issue didn't get resolved. Please provide the fix.

The following is the code for the config file for the custom dataset

dataset_type = 'MyDataset'
# Correct path of your dataset
data_root = 'data/my_dataset'

img_norm_cfg = dict( # This img_norm_cfg is widely used because it is mean and std of ImageNet 1K pretrained model
    mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)

crop_size = (512, 512) # Crop size of image in training

train_pipeline=[]
test_pipeline=[]
data = dict(
    samples_per_gpu=4, # Batch size of a single GPU
    workers_per_gpu=0, # Worker to pre-fetch data for each single GPU
    train=dict( # Train dataset config
        type=dataset_type, # Type of dataset, refer to mmseg/datasets/ for details.
        data_root=data_root, # The root of dataset.
        img_dir='img_dir/train', # The image directory of dataset.
        ann_dir='ann_dir/train',  # The annotation directory of dataset.
        pipeline=train_pipeline), # pipeline, this is passed by the train_pipeline created before.
    val=dict( # Validation dataset config.
        type=dataset_type,
        data_root=data_root,
        img_dir='img_dir/val',
        ann_dir='ann_dir/val',
        pipeline=test_pipeline), # Pipeline is passed by test_pipeline created before.
    test=dict(
        type=dataset_type,
        data_root=data_root,
        img_dir='img_dir/val',
        ann_dir='ann_dir/val',
        pipeline=test_pipeline))

The following is the python configuration file of the intended SETR model based on SETR_MLA.

_base_ = [
    '../_base_/models/setr_mla.py',
    '../_base_/datasets/my_dataset_config.py', '../_base_/default_runtime.py',
    '../_base_/schedules/schedule_80k.py'
]

# model settings
norm_cfg = dict(type='SyncBN', requires_grad=True)
model = dict(
    type='EncoderDecoder',
    backbone=dict(
        type='VIT_MLA',
        model_name='vit_large_patch16_384',
        img_size=512,
        patch_size=16,
        in_chans=3,
        embed_dim=1024,
        depth=24,
        num_heads=16,
        num_classes=3,
        drop_rate=0.1,
        norm_cfg=norm_cfg,
        pos_embed_interp=True,
        align_corners=False,
        mla_channels=256,
        mla_index=(5, 11, 17, 23)
    ),
    decode_head=dict(
        type='VIT_MLAHead',
        in_channels=1024,
        channels=512,
        img_size=512, 
        mla_channels=256,
        mlahead_channels=128,
        num_classes=3,
        norm_cfg=norm_cfg,
        align_corners=False,
        loss_decode=dict(
            type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0)))
# model training and testing settings
train_cfg = dict()
test_cfg = dict(mode='whole')
optimizer = dict(lr=0.002, weight_decay=0.0,
                 paramwise_cfg=dict(custom_keys={'head': dict(lr_mult=10.)})
                 )
img_norm_cfg = dict(
    mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
crop_size = (512, 512)
test_cfg = dict(mode='slide', crop_size=crop_size, stride=(512, 512))
find_unused_parameters = True
data = dict(samples_per_gpu=1)
test_pipeline=[]

data = dict(
    val=dict(pipeline=test_pipeline),
    test=dict(pipeline=test_pipeline))
 

It looks like this paper use ViT https://arxiv.org/abs/2010.11929 as backbone, with a simple decoder for segmentation?

To be honest it should be just an ablation study from ViT, instead of proposing a new paper.

If I am wrong please correct

Hi,

I downloaded a SETR_MLA model (512x512 with a batch size of 8) to test its performance using ADE20K validation set. Since I have only two RTX Titan, I replaced the #GPU 8 with 2 for testing.

The SS testing is same as its report, which is 47.79%.

For MS testing, I followed the instruction that uses the config file ending in _MS.py in configs/SETR. However, the MS testing is just 47.91%, which is far way lower than its report 50.03%.

So, I'm wondering what could be the possible reasons? Thanks.

"paramwise_cfg=dict(custom_keys={'head': dict(lr_mult=10.)}"

Hi, thank you for open-source your code firstly. I have a question about the configuration of the optimizer. I found there is "decode_head" in your model, not "head" used in 'custom_keys'. Will 'lr_mult=10' takes effect while we training the model?

Thanks~