Could you please comment on difference between feature usage in DeformableDETR and DeformableDETRsegm?

The first uses all features elements, and the latter only features[-1]...

Thank you!

https://github.com/megvii-research/SOLQ/blob/5471f58/models/deformable_detr.py#L136-L162 :

        features, pos = self.backbone(samples)

        srcs = []
        masks = []
        for l, feat in enumerate(features):
            src, mask = feat.decompose()
            srcs.append(self.input_proj[l](src))
            masks.append(mask)
            assert mask is not None
        if self.num_feature_levels > len(srcs):
            _len_srcs = len(srcs)
            for l in range(_len_srcs, self.num_feature_levels):
                if l == _len_srcs:
                    src = self.input_proj[l](features[-1].tensors)
                else:
                    src = self.input_proj[l](srcs[-1])
                m = samples.mask
                mask = F.interpolate(m[None].float(), size=src.shape[-2:]).to(torch.bool)[0]
                pos_l = self.backbone[1](NestedTensor(src, mask)).to(src.dtype)
                srcs.append(src)
                masks.append(mask)
                pos.append(pos_l)

        query_embeds = None
        if not self.two_stage:
            query_embeds = self.query_embed.weight
        hs, init_reference, inter_references, enc_outputs_class, enc_outputs_coord_unact, _, _ = self.transformer(srcs, masks, pos, query_embeds)

https://github.com/megvii-research/SOLQ/blob/5471f58/models/segmentation.py#L49-L55 :

        features, pos = self.detr.backbone(samples)

        bs = features[-1].tensors.shape[0]

        src, mask = features[-1].decompose()
        src_proj = self.detr.input_proj(src)
        hs, memory = self.detr.transformer(src_proj, mask, self.detr.query_embed.weight, pos[-1])

Hi, I am keeping focusing on DETR related work. This work is really interesting. It can be a good supplement answer to the issue https://github.com/facebookresearch/detr/issues/163.

Here, I am still have some questions:

1. I am surprised that UQR could provide more than 7 points improvement on AP^seg, as shown in Table2. May I ask for the detailed comparision about AP^seg_S,AP^seg_M and AP^seg_L? I wonder where the main improvement comes from?
2. As seen in table 1, SOLQ performs best on AP^box_L(much more better than other methods). I am confused that why its performance is not best on AP^seg_L. Could I regard it as the drawback of the mask compression coding on large objects? I wonder whether SQR could perform well on large objects and UQR performs better on small and medium objects. What's your idea?
3. If my understand correctly, this method can easily apply to panoptic segmentation. May I ask for some results about panoptic segmentation? With this result, we could better analyse the gap between SOLQ and DETR.

Version checking is buggy :(

https://github.com/facebookresearch/detr/commit/b9048ebe86561594f1472139ec42327f00aba699

Alternatively, all that legacy code can be removed and an assert added: from packaging import version; assert version.parse(torchvision.__version__) >= version.parse('0.8')

As the SOLQ is built on D-DETR, have you performed the experiment on the original DETR model? I would be very grateful if you can provide the results.

Hi,

You mentioned:

Higher performance (Box AP=56.5, Mask AP=46.7) is reported by training with long side 1536 on Swin-L backbone, instead of long side 1333.

May I know the image resizing strategies during training and testing? I found some commented codes for Swin-L, for training https://github.com/megvii-research/SOLQ/blob/main/datasets/coco.py#L135 and for testing https://github.com/megvii-research/SOLQ/blob/main/datasets/coco.py#L158. But for testing the long side is 1333 instead of 1536. Could you please clarify this? Thank you very much!

1. Do I understand correctly that SOLQ(..., with_vector = False) is equivalent to vanilla DeformableDETR(...)?
2. Why are postprocessors created only in args.eval? https://github.com/megvii-research/SOLQ/blob/b35360390b1d51d375dd9d03c39dbce663e223a7/models/solq.py#L614 The postprocessors should be applied at regular evaluation during training too, right? In fast_solq code only args.masks is checked: https://github.com/megvii-research/SOLQ/blob/b35360390b1d51d375dd9d03c39dbce663e223a7/models/fast_solq.py#L588. What's the motivation for this difference?

Thanks!

Hi, authors, have you tried computing the matching cost considering cls, reg, and mask together ? Is there any difference in the performance? Thanks in advance.

Error

ImportError

Steps to reproduce the behavior:

1. Git clone the repository of SOLQ
2. Update the dataset you want to use.
3. Update the data paths in the file SOLQ/datasets/coco.py
4. RUn the bash file configs/r50_solq_train.sh

Expected behavior

It should now show the error and move further to run the SOL-Q model.

Environment

PyTorch version: 1.9.0+cu102 Is debug build: False CUDA used to build PyTorch: 10.2 ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64) GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final) CMake version: version 3.12.0 Libc version: glibc-2.26

Python version: 3.7.11 (default, Jul 3 2021, 18:01:19) [GCC 7.5.0] (64-bit runtime) Python platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic Is CUDA available: True CUDA runtime version: 11.0.221 GPU models and configuration: GPU 0: Tesla T4 Nvidia driver version: 460.32.03 cuDNN version: Probably one of the following: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5 /usr/lib/x86_64-linux-gnu/libcudnn.so.8.0.4 /usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.0.4 /usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.0.4 /usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.0.4 /usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.0.4 /usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.0.4 /usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.0.4 HIP runtime version: N/A MIOpen runtime version: N/A

Versions of relevant libraries: [pip3] numpy==1.19.5 [pip3] torch==1.9.0+cu102 [pip3] torchsummary==1.5.1 [pip3] torchtext==0.10.0 [pip3] torchvision==0.10.0+cu102 [conda] Could not collect

Additional context

When running the script !bash configs/r50_solq_train.sh it, shows ImportError like shown below :

Traceback (most recent call last):
  File "main.py", line 22, in <module>
    import datasets
  File "/content/SOLQ/datasets/__init__.py", line 13, in <module>
    from .coco import build as build_coco
  File "/content/SOLQ/datasets/coco.py", line 23, in <module>
    from util.misc import get_local_rank, get_local_size
  File "/content/SOLQ/util/misc.py", line 36, in <module>
    from torchvision.ops.misc import _NewEmptyTensorOp

Hi authors,

Thanks for the excellent work! I wish to extend SOLQ to panoptic segmentation. Shall I simply treat stuff and thing equally? Looking forward to your suggestions!

Hi there, I'm trying to visualize the mask after idct, the code is:

save_dir = './vis'
gt_mask_len = 512
n_keep = 128
processor_dct = ProcessorDCT(n_keep=n_keep, gt_mask_len=gt_mask_len)
#mask = cv2.imread(img_path, 0).astype(np.float32)
mask = np.load('mask.npy')
new_mask = np.array((mask==1)).astype(np.float32)

new_mask = cv2.resize(new_mask, (gt_mask_len, gt_mask_len))
coeffs = cv2.dct(new_mask)
cv2.imwrite(os.path.join(save_dir, '{}_coeffs.png'.format(name.split('.')[0])), coeffs)

idct = np.zeros((gt_mask_len**2))
vectors = torch.from_numpy(coeffs).flatten()
vectors = vectors[torch.tensor(processor_dct.zigzag_table)]

idct[:n_keep] = vectors.cpu().numpy()
idct = processor_dct.inverse_zigzag(idct, gt_mask_len, gt_mask_len)
cv2.imwrite(os.path.join(save_dir, '{}_i_coeffs.png'.format(name.split('.')[0])), idct)
re_mask = cv2.idct(idct)
max_v = np.max(re_mask)
min_v = np.min(re_mask)
re_mask = np.where(re_mask>(max_v+min_v) / 2., 255, 0)
cv2.imwrite(os.path.join(save_dir, '{}_recover.png'.format(name.split('.')[0])), re_mask)


plt.figure(1)
plt.imshow(new_mask)
plt.figure(2)
plt.imshow(re_mask)

new_mask shows:

re_mask shows:

The re_mask looks different with the original mask.

I was wondering why this is and if there is anything I am doing wrong.

“Thanks for your attention on SOLQ! It will take about 1.5 days and 2.0 days to train SOLQ with R50 and R101 backbones, respectively. As for the Swin-Large backbone, it will take nearly four days to train due to the large computation cost.”

Excuse me, do you use 2 GPUs or 8 GPUs for training for 1.5 days? the read me you provided uses 8 GPUs. so I confused

Hello,

First of all, thank you for your great work.

I want to ask a point in the implementation of input projection before the MaskHeadSmallConv in segmentation.py . The implementation applies stride 2 to the features which makes the best stride 8. However, for the segmentation tasks, it is possible to get better result when the stride 4 is utilized for the mask creation. The original segmentation head implementation of DETR also utilizes in that way. Therefore, I want to ask that what is the reason for utilizing that stride in your implementation?

Thanks in advance

Hi @dbofseuofhust, @vaesl!

Can't find in the code the URLs to the ImageNet-pretrained Swin-L. Which checkpoints did you use? https://github.com/microsoft/Swin-Transformer provides many different ones.

Could you please publish a config for training using Swin-L?

Are your modifications to swin_transformer.py upstreamed anywhere?

I also wonder, have you tried other Swin backbones like Swin-S or Swin-B? ESViT repo publishes some self-sup trained Swin, but they are only for Swin-S/T/B: https://github.com/microsoft/esvit ...

Thank you!

Hi!

Would you have any guidance / code on reproducing the decoder attention visualization? (fig. 4 from A.3). I'm worried of making some padding-related mistakes while working with sampled_locations

Thanks!