Hello,

When I tried num_feature_levels > 1, the code won't work. This line srcs = torch.cat(srcs, dim=1) in anchor_detr.py shows the mismatch tensor size error. Any ideas to fix it?

Hi, AnchorDetr massively fixed detr problem on slow converge speed as well as boosted AP on small.

However, notice that AnchorDETR using 900 as output candidates (num queries), this might not a problem in detection, but once extend it into panoptic segmentation, 900 become a problem. The mainly reason are:

1. Panoptic in detr treat stuff and things all in things, which means, it will output a gaint tensor: 900x400x400 for example for every possible instances and semantic seg in image, that is a very huge tensor if input resolution not small;
2. 900 might accelerate training on detection, but not for instance seg or semantic seg.

Does there any exp on this or how to make it possible do panoptic seg in anchordetr way? Any suggestion would be very appreciated.

Hi, thanks for your great work. When I load the anchor_detr_r50_dc5.pth into the model and training on other dataset. The training loss is reducing, but the test loss remains around the same value in the 30 epoches. Do you have any experience with this? I reduced the number of reference points and the number of patterns, but still the same problem.

When training AnchorDETR with images per batch 8, and learning rate 0.0001, it keeps getting nan error, such as

generalized_box_iou assert (boxes1[:, 2:] >= boxes1[:, :2]).all(), f"incorrect boxes, boxes1 {boxes1}" AssertionError: incorrect boxes, boxes1 tensor([[nan, nan, nan, nan], [nan, nan, nan, nan], [nan, nan, nan, nan], ..., [nan, nan, nan, nan], [nan, nan, nan, nan], [nan, nan, nan, nan]], device='cuda:4')

Any idea to fix this issue?

the onnx model exported has very wierd dimension caused it can not be simplifed or pass onnx.checker.check.

This is verbose output of export DETR:

this is verbose output of AnchorDETR:

Both are last serveral layers, as you can see, for DETR the strides seems very small

but AnchorDETR are something like Float(1, 1, 900, 91, strides=[81900, 81900, 91, 1], requires_grad=1, device=cuda:0) gaint value.

and it caused when try to check this model, or try to simplifed this model:

ValueError: Message onnx.ModelProto exceeds maximum protobuf size of 2GB: 3714028571

error got.

any idea?

Hi,

I am studing your shared code using a custom dataset.

Here are key points:

• I splitted my dataset in train, valid and test
• I trained for 100 epochs (I saw recommendation to run it for 50 epochs) using AnchorDETR-DC Resnet 101 weights for transfer learning.

I would like to understand why after 50 epochs the performance of my train starts get worse (overfitting?).

Here are the graph:

Run a validation code under my test dataset I got the following results using the saved weights:

Saved weights of epoch: 0049
IoU metric: bbox
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.599
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.847
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.637
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.516
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.624
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.632
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.677
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.703
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.616
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.727

Saved weights of epoch: 0099
IoU metric: bbox
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.558
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.806
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.607
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.426
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.594
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.620
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.653
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.663
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.504
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.702

Why the results on epoch 49 is better than results on epoch 99? Is there some option to save only the best weights?

using bs=80, the AP at 30 can not goes up anymore:

what could be the reason?

The model, the Criterion all same as AnchorDETR, all loss weights are same....

but the performance so bad.... Even it indeed converge fast at the first serveral iterations....

Hi, I'm very interested in your work about the newly decoder and self-attention of DETR，but I have some questions about your code. 1. when I debug the code, I found a detail not mentioned in the paper：when use RCDA attention, it is divided into two situations, like this : if efficient_compute: if src_len_col<src_len_row: b_ein,q_ein,w_ein = attn_output_weights_row.shape b_ein,h_ein,w_ein,c_ein = v.shape # Ax * V得到论文中的Z [bs*8,HW,H*32] -> [bs*8, H, HW, 32] attn_output_row = torch.matmul(attn_output_weights_row,v.permute(0,2,1,3).reshape(b_ein,w_ein,h_ein*c_ein)).reshape(b_ein,q_ein,h_ein,c_ein).permute(0,2,1,3) # Ay * Z得到论文中的attention outputs: [HW,8,bs,H]*[HW,bs*8,H,32] -> [HW,8,bs,32] -> [HW,8*bs,32] -> [HW, bs, 256] attn_output = torch.matmul(attn_output_weights_col.permute(1,0,2)[:,:,None,:],attn_output_row.permute(2,0,1,3)).squeeze(-2).reshape(tgt_len,bsz,embed_dim) ### the following code base on einsum get the same results # attn_output_row = torch.einsum("bqw,bhwc->bhqc",attn_output_weights_row,v) # attn_output = torch.einsum("bqh,bhqc->qbc",attn_output_weights_col,attn_output_row).reshape(tgt_len,bsz,embed_dim) else: b_ein,q_ein,h_ein=attn_output_weights_col.shape b_ein,h_ein,w_ein,c_ein = v.shape attn_output_col = torch.matmul(attn_output_weights_col,v.reshape(b_ein,h_ein,w_ein*c_ein)).reshape(b_ein,q_ein,w_ein,c_ein) attn_output = torch.matmul(attn_output_weights_row[:,:,None,:],attn_output_col).squeeze(-2).permute(1,0,2).reshape(tgt_len, bsz, embed_dim)

Why is there two cases to calculate RCDA Attention outputs, in the code， if col < row, Z = Ax * V，then outputs = Ay * Z； if col > row, Z = Ay * V, then outputs = Ax * Z。in my opinion， when col < row, the attention outputs = Ay * Ax * V , otherwise the outputs = Ax * Ay * V，What is the reason for this design?

2. RCDA attention module is only used in Self-Attention of Encoder and Cross-Attention of Deocder，Why not use it in Self-Attention of Decoder？ I sincerely hope I can get your answers！ Thanks ！

Hi, Thank you for your good work!

Recently i am studying DETR-like, and i have questions as following:

1.there is no implement about reference point iterative refinement , but many other works do that and it seems work well. I want to know have you tested that? And i notice that your work (Anchor-DETR) which had been mentioned in DAB is implemented by refinement.

2.i am confusing about the word "object query", it seems different meanings in diiferent works. I want to know, in your opinoin, what the "object query" refers to? The content part(tgt in code), or positional part, or tgt + positional part.

3.Which part do you think is more important? content part or positional part

thank you again.

博主你好，我最近在复现您的模型，根据你的readme文件，下载了模型参数，并运行了eval的代码 1.python main.py --eval --coco_path /data/coco --resume /weights/AnchorDETR_r50_c5.pth 但是发现选择eval模式进行测试的时候，并不会使用coco的测试数据集，而是使用的coco的训练集进行测试。

Hello thank you for sharing the code.

I would like to know how to extract precision, recall and f1-score metrics. I already have the AP and AR metrics.

I am trying to use the following code but it gives me a numpy matrix:

precision = coco_eval.eval['precision']
recall = coco_eval.eval['recall']

Can you help me?