Hello.I really aprreciate for your project.

However, The following error occurs when a 512-size image is input at Attention class.

dots = dots + relative_bias RuntimeError: The size of tensor a (1024) must match the size of tensor b (196) at non-singleton dimension 3.

https://github.com/chinhsuanwu/coatnet-pytorch/blob/d3ef1c3e4d6dfcc0b5f731e46774885686062452/coatnet.py#L155

Why this error is occured? How do I edit your code when I want to resize the image?

Thank you!

Hi, I can't found the code about stochastic depth in your implementation.

And I add the stochastic depth code and train a CoAtNet-Tiny on ImageNet 1k, but got 79.27%@top1.

Have you reproduce the results reported by the paper?

Hi, I noticed that the value of self.relative_bias_table is always all 0, then the following: relative_bias = self.relative_bias_table.gather( 0, self.relative_index.repeat(1, self.heads)) is actually meaningless (it is all 0)? Thanks!

Hey, I tried with your implementation, and I found the calculated #param is a little bit different from the paper, and I am curious about the reason, could you please help me out?

Take coatnet_0 for example, the calculated result is 17789624 ( 17789624 / 2^20 = 16.97), and the reported #param of the paper is 25M

Thanks in advance

Thanks for your sharing. We want to confirm that the relative_coords is learnable parameters or constant in CoatNet?

        coords = torch.meshgrid((torch.arange(self.ih), torch.arange(self.iw)))
        coords = torch.flatten(torch.stack(coords), 1)

        relative_coords = coords[:, :, None] - coords[:, None, :]
        relative_coords[0] += self.ih - 1
        relative_coords[1] += self.iw - 1
        relative_coords[0] *= 2 * self.iw - 1
        relative_coords = rearrange(relative_coords, 'c h w -> h w c')
        relative_index = relative_coords.sum(-1).flatten().unsqueeze(1)
        self.register_buffer("relative_index", relative_index)

Hi,

I tried to train CoAtNet_0 with tiny image net from cs231n (200 classes). Seems the model does not converge.

Could it be that the implementation is not 100% correct? For example, the positional embedding indexing part. I went through the code and I think other components should be correct.

Except for the pos embedding indexing, I'm not good enough to comprehend it. Do you have a reference for the implementation of the positional embedding indexing part?

Hello, first off, really appreciate your work! Unfortunately, I'm getting overfitting using a custom dataset even in coatnet_0, is there a workaround? Please! Thank you veru much!

Hello,

I'm training the model from scratch on my custom dataset and the convergence is very slow. So, can you share any pretrained networks if possible?