Hi, thanks for releasing the code. When I reproduce the table 6 Imagenet-C result in your paper (severity level 5), the source error is 82.4 and the BN adapt is 72.1, which is the same as your paper. However, Tent result is 69.2 and the CoTTA result is about 71.6, which is inconsistent with your paper, in which the tent is 66.5 and CoTTA is 63. Could you explain why the result is different? Thanks

Awesome work, but a little question:

Therefore, we use a threshold $p_{th}$ to filter the images, and do not apply augmentations on those with high confidence. More specifically, we design $p_{th} = conf^S-\delta$, where $conf^S$ is the 5% quantile for the softmax predictions’ confidence on the source images from the source model $f_\theta$

In your code, 0.74 was the 5% quantile for cityscapes. So ,how is this value calculated?

Kind regards.

The error is "_pickle.UnpicklingError: invalid load key, '<'." when checkpoint = torch.load(model_path, map_location=torch.device('cpu')), line 127 in utils.py is executed. The reported size of Standard.pt is 2.2K. Is this file complete?

Hi, For some reason, I can't use request to download the base model from Google Drive, but I can download it by entering the Google Drive link from Chrome, and the link is "https://docs.google.com/uc?export=download&id=1t98aEuzeTL8P7Kpd5DIrCoCL21BNZUhC". The name of the file in the link is "natural.pt.tar", how do I change it to "Standard.pt"? Can I rename it directly? Hope to get your help.

Hi Qin, I encounter an error (Pip failed) while creating the environment for segmentation "Segformer" using conda, is it possible to re-write your environment locally again ? i.e. on your local workstation, not a cluster?

Hi! Thanks for sharing your code. This work is interesting! I have some questions about the updating of the teacher model. It would be great if you could answer them. The original Mean Teacher is trained with both labeled and unlabeled data. The consistency loss then calculates the difference between the outputs from the student and the teacher model. The author did not have high confidence about the feasibility of training Mean teacher without any labeled data. https://github.com/CuriousAI/mean-teacher/issues/24
In your code, the student model update with the softmax_entropy and the teacher model accumulated the historical weights from the student model. I am wonder how could you avoid such performance degradation on the teacher model when there is no other guidance on the student model. I have tried to implement the Mean Teacher model without labeled data training on the student model. The training results shows that the performance of the teacher model dropped when updated with exponential moving average every steps. Had you ever encountered such problems?

Hi,

Thanks for patiently answering all the questions in previous issues. The paper is interesting, congratulations! I had a few questions and it would be great if you could help me understand this better :

1. Which model is being used for evaluation? The model running on severity 5, has it already learnt/updated from all previous severities? When you move from one corruption to next while evaluating, has the model learnt from the last one or has it been tested individually on each corruption?
2. In each batch, is there only one severity and one corruption?

Dear author, thank you for exposing the code. But when I want to reproduce the code of segmentation, I can't open the URL you provided. I hope you can answer my doubts. Thank you again!

Hi, I have a query regarding the batch size. In the paper you have written, "We update the model for one step at each iteration (i.e. one gradient step per test point).". This seems to suggest that the BATCH_SIZE = 1.

But in the config file, it is BATCH_SIZE = 200. [See here: link] So is the actual batch size 200 (and not 1) for the experiments in your paper?

I have tried setting the batch size to 1, but the results deteriorate drastically.

A clarification would be helpful. Thank you.

Thanks for yout work, which inspired me a lot. And this work is highly corresponding to the continual learning or lifelong learning. Have you compared with other continual learning methods?

Hi, author! Thanks again for your sharing codes. I've fixed the problems occurring in reproducing CIFAR-10C results by replacing the model back to the standard one. And I am here for some questions in your proposed restoration design. We know that "Stochastic Restoration" in CoTTA is designed to avoid catastrophic forgetting. The thing that bothers me is, in CL, avoiding catastrophic forgetting is needed when you want a model to solve all the tasks/domains that have been seen previously. But when all you need is just to keep high performance in current tasks/domains (i.e., online performance), what might be the reason for choosing restoration to avoid catastrophic forgetting (i.e., keep previous knowledge)? I know it achieves better results, and as the paper mentioned, it helps recovery after encountering hard examples. And I'll still appreciate it very much if there is any answer for my puzzles (it is also totally okay not to do so :) ). Or maybe is there any advice for the restoration factor setup (why CoTTA sets 0.01) ? :)