Hi,

Thank you for the wonderful paper. I have trained the CCT_7/3x1 by default and however, the result is differ from the paper. CIFAR10: 93.67 CIFAR100: 73.15 I've tried it many times and It never reaches 94.72% for CIFAR10 and 76.67% for CIFAR100. Could you please help me what makes it differ?

Thanks.

Hi @alihassanijr,

Many thanks for your super interesting work, and sharing the elegant code with the community.

I am able to replicate your CIFAR-10 and CIFAR-100 results perfectly. But, there is a large gap when it comes to the Flowers dataset.

After running the following command:

python train.py -c configs/datasets/flowers102.yml --model cct_7_7x2_224_sine ./data/flowers102 --log-wandb

I am able to get only 62% accuracy. Please find the wandb report here. I am attaching the logs too:
output.log

The only change that I made to the code was to use the PyTorch dataloaders:

from torchvision.datasets import Flowers102
dataset_train = Flowers102(root=args.data_dir, split="train", download=True)
dataset_eval = Flowers102(root=args.data_dir, split="test", download=True)

I am sure that this might be some minor configuration issue for Flowers Dataset, as I am able to replicate the results on CIFAR-10 and CIFAR-100.

Thanks again, and it would be very kind of you if you could help me.

Thanks, Joseph

Hello, I got the error "AttributeError: 'TransformerClassifier' object has no attribute 'num_tokens'" when I use cct_7_7x2_224_sine and set pretrained=True

Hi @stevenwalton

What is the difference between your "TransformerEncoderLayer" and original Vit paper "Transformer" class?

Orginal VIT

class Transformer(nn.Module):
    def __init__(self, dim, depth, heads, dim_head, mlp_dim, dropout):
        super().__init__()
        self.layers = nn.ModuleList([])  # There are using Residual
        for _ in range(depth):
            self.layers.append(nn.ModuleList([
                Residual(PreNorm(dim, Attention(dim, heads=heads, dim_head=dim_head, dropout=dropout))),
                # Here they implemented Residual
                Residual(PreNorm(dim, FeedForward(dim, mlp_dim, dropout=dropout)))
            ]))

    def forward(self, x, mask=None):
        for attn, ff in self.layers:
            x = attn(x, mask=mask)  # Chnage in this part
            # embed()
            x = ff(x)
        return x

Your Transformer

class TransformerEncoderLayer(nn.Module):
    """
    Inspired by torch.nn.TransformerEncoderLayer and
    rwightman's timm package.
    """

    def __init__(self, d_model, nhead, dim_feedforward=2048, dropout=0.1,
                 attention_dropout=0.1, drop_path_rate=0.1):
        super(TransformerEncoderLayer, self).__init__()
        self.pre_norm = nn.LayerNorm(d_model)
        self.self_attn = Attention(dim=d_model, num_heads=nhead,
                                   attention_dropout=attention_dropout, projection_dropout=dropout)

        self.linear1 = nn.Linear(d_model, dim_feedforward)
        self.dropout1 = nn.Dropout(dropout)
        self.norm1 = nn.LayerNorm(d_model)
        self.linear2 = nn.Linear(dim_feedforward, d_model)
        self.dropout2 = nn.Dropout(dropout)

        self.drop_path = DropPath(drop_path_rate) if drop_path_rate > 0 else nn.Identity()

        self.activation = F.gelu

    def forward(self, src: torch.Tensor, mask=None, *args, **kwargs) -> torch.Tensor:
        src = src + self.drop_path(self.self_attn(self.pre_norm(src)))
        src = self.norm1(src)
        src2 = self.linear2(self.dropout1(self.activation(self.linear1(src))))
        src = src + self.drop_path(self.dropout2(src2))
        return src

Actually, I made some modifications in the original VIT and I want to add that modified part in your Transformer Encoder layer but your's code is different in terms of TransformerEncoderLayer.

Hi,

I'm trying to figure out how to train your model to achieve the SOTA accuracy you report on Flowers102. It seems like using finetuned/cct_14-7x2_flowers102.yml will download the model with the 99.76% test accuracy you report, but I can't find any config files which actually train this model from scratch (or from e.g. an ImageNet checkpoint if you use that). Do you mind pointing me to any config files for this that I might have missed, or else to a description of the training procedure for your SOTA Flowers102 model so that I can try to reproduce it?

Thanks for your help, Will

Hi, can you share the HP that your team trained it on RandAug with Timm? I have followed the same steps but still stuck on 76.9% and couldn't go any further (trained for 300 epochs should reach approximately 80%). Thanks for your help.

Thank you for sharing this amazing work. I am currently attempting to apply your ideas to a specific problem with bigger images sized 128x128. Do you have any recommendations on how to improve the performances of your network on bigger images?

Hi, thank you for this very clean open-source implementation!

I've been testing out some modifications and noticed, even without the modifications, I'm not quite achieving the same accuracies as you report on CIFAR-10. I wondered if you had any suggestions about things that I might have missed.

Specifically, I tried to reproduce your results using one of your configs with the command python train.py -c configs/pretrained/cct_7-3x1_cifar10_300epochs.yml --model cct_7_3x1_32 datasets/CIFAR-10-images/ --log-wandb. I copied the dataset from this github repo. I couldn't find details on whether you use a train/validation split so then trained on all 50000 training images (i.e. with no validation set) and tested on all 10000 test images. I used your validate function for computing the test accuracy (by renaming the test image folder so that it is loaded as a validation set). After the full 300 epochs, I obtained 93.21% test accuracy, rather than the 96.53% that I think you report for this config in the README. Please let me know if there's anything I should do differently to obtain these results - perhaps using a different train/test split, computing the test loss in a different way, turning on EMA averaging, or if there's anything else that I might have missed.

I also tried computing test statistics in the same way after loading your pretrained cct_7_3x1_32 checkpoint (instead of training it) and got a lower test accuracy of 91.67%. So this makes me think that the issue is likely related to testing rather than training.

Thanks!

I want to reproduce your paper, however I find the weight decay and learning rate of cifar10 in *.yaml is different from paper's said, could your please tell me which parameter should I use?

How many image reductions should I have? Is this right?

model = cct_7(img_size=im_size,
              num_classes=classes,
              positional_embedding='learnable',
              n_conv_layers=2,
              kernel_size=7,
              stride=2,
              padding=3,
              pooling_kernel_size=3,
              pooling_stride=2,
              pooling_padding=1)

Hi, sorry to trouble you again. I have successfully your results on cifar10, and I plan to reproduce the results on ImageNet. I follow your configs for imagenet and my result is lower than yours about 2%.

I would like to know the interpolation's type used in your paper for imagenet. As most paper's only use Bicubic, and your paper use "random" for imagenet.

Thanks very much!

Hi, this work is awesome. I just have one little question. The paper says the total batch size is 128 for CIFAR's and 4 GPU's were used in parallel. That doesn't mean the total batch size is 128 * 4 = 512, does it? DDP is for Imagenet, and non-distributed is for CIFAR, am I correct?

Thanks a ton :)

Hi,

i am a little confuse about the output of the CCT. If I have a classification task with n possible classes, are the outputs the logits of each class? Thus to find the respective probabilities i have to apply a softmax function, or are the outputs the probabilities?

Thanks in advance

Hi,

There was a small problem with the mask returned from TextTokenizer forward function. The next function using this mask needs a 2D tensor. Therefore, in TextTokenizer, the mask should not be unsqueezed before being returned.

The problem is fixed in this pull request.

Hello,

It seems to make more intuitive sense to use 1D convolutions here over the embedding with a channel size equal to the word embedding dimension, rather than the edge-case of a 2D convolution as is currently implemented. I would personally make this change to match other networks with similar convolutions over nn.Embeddings. I believe this has no change to performance but rather is presented for clarity. Thank you

First of all, thanks for your amazing work!

And it seems that your TransformerEncoderLayer implementation is a bit different from the 'mainstream' implementations, because you create your residual link after the LayerNorm procedure:

https://github.com/SHI-Labs/Compact-Transformers/blob/3f3d093746bc58213d9e9af4431242d305717855/src/utils/transformers.py#L96-L99

However, from the original paper of ViT and many other implementations, the residual link is created before the LayerNorm procedure:

src = src + self.drop_path(self.self_attn(self.pre_norm(src)))
src2 = self.norm1(src)
src2 = self.linear2(self.dropout1(self.activation(self.linear1(src2))))
src = src + self.drop_path(self.dropout2(src2))

I'm just wondering whether this is on purpose or some kind of 'typo'? Thanks in advance!