Hello! First of all, thank you for sharing your great work :) I've tested your code on CIFAR-10-C (severity 5) using CIFAR-10 pretrained ResNet 18/34/50, following the configuration in /master/cfgs/tent.yaml. While the tent indeed increases the classification performance at the very first epoch, it deteriorates with more epochs (even below the baseline model). This is my result:

Models | Baseline (%) | Tent (%) (epoch 1/2/3) ------------|------------|------------ ResNet18 | 62.23 | 78.85 / 77.02 / 74.33 ResNet34 | 62.43 | 74.31 / 64.24 / 53.74 ResNet50 | 61.84 | 67.58 / 47.56 / 34.82

I expected the performance to gradually increase, but it didn't. Is this result reasonable, or did I do something wrong? Thank you :)

As the paper said, TNET needs 2× the inference time plus 1× the gradient time per test point, but I found there is only one forward and one gradient update in the code https://github.com/DequanWang/tent/blob/03ac55c4fef0fda3eacb2f6414b099031e96d003/tent.py#L49

Is the right order: forward, backward, and forward?

Hi @DequanWang ,

Thanks for you open source the code firstly, I try to use the model as my paper baseline, but the training loss value is always 0.

My training script code following,

self.model.train()
self.model = tent.configure_model(self.model)
params, _ = tent.collect_params(self.model)
self.optimizer = optim.Adam(params, lr=3e-4)
tent_model = tent.Tent(self.model, self.optimizer)

for idx, data in enumerate(self.train_loader):
        image = data['img'].type(torch.FloatTensor).cuda(non_blocking=True)

        output, loss = tent_model(image)

It's worth mentioning that this happens whether I load the model or not.

Looking Forward to Hearing from You.

I have a simple question related to the offline tent experiment. What should I change if I want to get results of the offline tent (ex, Table 2)?
Should I change _C.MODEL.EPISODIC = True https://github.com/DequanWang/tent/blob/e9e926a668d85244c66a6d5c006efbd2b82e83e8/conf.py#L37-L39

Termination For online adaptation, no termination is necessary, and iteration continues as long as there is test data. For offline adaptation, the model is first updated and then inference is repeated. Adaptation may of course continue by updating for multiple epochs.

all i know is minimizing the shannon entropy can lower the uncertainty of the model, but how can i assure the model is becoming more accurate instead of becoming less accurate after minimizing the entropy?looking forward to an answer

Hello! First of all, thank you for sharing your great work

When I run example--WRN-28-10, I find that: url_base ,which is = "https://docs.google.com/uc?export=download", can't be accessed. Is the link invalid？

If it is because the url is invalid, could you upload it again?

thank you very much!

Hi,

Thank you for a very nice paper. I have tried to reproduce (or get close) to the results mentioned in the paper. I cannot reproduce the results for Cifar 10c and Cifar 100c by using the resnet-26 as also used by TTT (Sun et. al) paper. I am pretty sure, that I am doing everything right because my trained (resnet-26) gives me an error of 8.2 percent on the clean test set.

When I try to use it with tent, the performance degrades. I am attaching the log file for Resnet 26 here with this issue just so you know what is happening. I must also mention that the results mentioned in the paper (for Cifar10c and 100c) were reproduced by using a Wide resnet 28. But there is NO mention of using a wide resnet 28 in the paper.

Did you update the final version of the paper or I am doing something wrong? I am also attaching the log file with this issue for the results obtained by using wide resnet 28.

Also, can you please tell me what architecture did you use for the imagenet results? Basically when I take an off the shelf resnet 50 (pretrained on imagenet) and try to reproduce the results. It doesn't. In fact the system totally blows up! Is it some other model used for imagenet results as well, instead of resnet 50? Also, adding the log file for Imagenet results.

I look forward to your answer.

Thanks and BR,

Mirza.

tent_resnet_26.txt tent_wide_resnet_28.txt tent_imagenet.txt

Hi, thanks for your great work. I re-implement Tent based on Transfer-Learning-Library to evaluate its performance on DA datasets. And I only update BN layers as recommented in the paper. However, I find the accuracy is really low. Could you please release the code to help reproducing the results on Digits and VisDA. Thanks a lot.