Greetings! I can only get 46.81% accuracy and 47.05% per-class accuracy after running bash zeroshot.sh stanford_cars rn50. However, the reported accuracy on StanfordCars dataset is ~55%. What's wrong?

First of all, thank you for open sourcing such an easy to use code :) I reproduced your reported results in CoOp on two datasets, DTD and Flower101. I ran the code with three random seeds,1,2 and 3 for both datasets, as your default setting in ./scripts/main.sh. The performance of model on DTD is as well as the result in paper (acc: 63.46) when trained with seed=3, but the results of seed 1 and 2 are poor (acc: ~15). As for Flower101, the result of seed 2 and 3 are ~94, but seed 1's result is 44.50

I wonder if this is a normal situation for this few shot training setting? Thanks for any suggestion :)

If the token prefixes and suffix are just the slice of the embedding, for example, replacing self.register_buffer("token_prefix", embedding[:, :1, :]) with self.token_prefix=embedding[:, :1, :]) in this line, we will not have to ignore those when loading. Therefore, why do token prefixes have to be a buffer type? Thanks a lot!

Hello, Thank you for sharing your great work.

I had a question regarding the few-shot setting in the CoCoOp experiments. In the paper, it is mentioned that CoCoOp follows a zero-shot evaluation (from base to novel classes) but for training the base classes, it uses a few-shot setting. However, generally for zero-shot evaluation, models are trained on the complete base classes.

Does this mean that, CoCoOp and CoOp requires only a few-shot setting to perform well on novel categories. Can the same training recipe of CoCoOp or CoOp be used by training all examples of the base classes?

Thank you and kind regards.

Hi, Thanks a lot for the excellent work and the easy-to-use code! Recently I've been trying to use CoOp and CoCoOp in my research. However I encounter a small problem: the GPU consumption of CoCoOp seems to be much larger (about 64X under my setting) than CoOp, resulting in small batch size and very long training time. Based on my understanding, the reason is that the prompts in CoCoOp should be given to each instance instead of each batch. I've seen the same problem reported in the paper. May I ask whether there are any tricks during training to accelerate the training process? Thanks so much!

Thanks for your contributions!

I have a question about "classnames = self.dm.dataset.classnames" (the line 224 of "CoOp.build_model" in coop.py). What is the value of "classnames"? I checked the configuration files and didn't find out.

Hi, may I ask if the gradients of the original CLIP text encoder are frozen or not? The paper mentioned that the gradients of text encoder is frozen, but I couldn't find that part in the code... Thanks a lot for your help!

Dear Kaiyang, I find a bug in the shells for CoCoOp. I think a correct way is to remove the top line "cd ../../" in the .sh scripts, or it will raise "train.py no such file" error. Thanks.

1. To my knowledge, CLIP can be directly used applied to zero-shot learning (i.e., unseen/novel classes). coop and cocoop don't appear to be zero-shot learning, but require fine-tuning. However, I don't see the detials about how to fine-tuning in paper. Am I misunderstand it? In the meantime, I would like to know how the CLIP is fine-tuned.
2. I cannot understand the figure 1 in paper: why the performance of coop and cocoop can be compared to zero-shot learning.

Thanks for your great work! I tried to use your code to reproduce some results of CoOp reported in your CoCoOp paper. I tried this model on the dtd dataset with: bash main.sh dtd vit_b16_ep50 end 4 16 False. Which is exactly same as the setting in the paper. I got a much higher performance: accuracy: 67.38% +- 0.51%. But the paper report CoOp performance as 54.24.

I have been successful in developing the train and test pipeline for my custom dataset. Can you help me out for making inference on a single image. I am using the trainer.model_inference(image) function. Is there a particular format this image needs to be in ? I am using PIL to read the image.

Error: /ContextOptimization/CoOp/trainers/coop.py", line 196, in forward image_features = self.image_encoder(image.type(self.dtype)) File "/home/chandan/anaconda3/envs/coop/lib/python3.8/site-packages/PIL/Image.py", line 519, in getattr raise AttributeError(name) AttributeError: type

Main function used:

def main(args): cfg = setup_cfg(args) if cfg.SEED >= 0: print("Setting fixed seed: {}".format(cfg.SEED)) set_random_seed(cfg.SEED) setup_logger(cfg.OUTPUT_DIR)

if torch.cuda.is_available() and cfg.USE_CUDA:
    torch.backends.cudnn.benchmark = True

print_args(args, cfg)
print("Collecting env info ...")
print("** System info **\n{}\n".format(collect_env_info()))

trainer = build_trainer(cfg)

trainer.load_model(args.model_dir, epoch=args.load_epoch)
image = Image.open('/ContextOptimization/CoOp/data/0cd2ed50.png')
result = trainer.model_inference(image)
print(result)
return result

I am looking for the predicted class and predicted probabilities as output.

Any direction would be appreciative.

Thanks

Thanks for your great work. I would like to ask you whether you have considered using a CNN network such as ResNet as the backbone in CoCoOp and whether it is possible to use it?

Thanks for your great work！ The previous issue has been solved， but I find a new issue. If I change the code (https://github.com/KaiyangZhou/CoOp/blob/ff61507c790454bce7c5052c3ac39e60772f1f89/trainers/coop.py#L248) as self.register_model("model", self.model, self.optim, self.sched). The result will drop considerably(20%)！ Can you give me some advice?

Hi, thanks for the nice code. I found the performance is poor when full fine-tuning the ResNet-based CLIP on ImageNet while for ViT-based CLIP the performance is good. Do you have some insightful comments on why full fine-tuning or linear probing the ResNet-based CLIP makes the performance worse?

Thanks for your great job! I want to ask why the input is not (image, text) at forward function, such as output = self.model(image, text) . And what is the scheme of matching text logits and image logits?

Dear Zhou: When I train my network on oxford_flower(epoch=200), it get a great result as follows: => result

• total: 2,463
• correct: 2,268
• accuracy: 92.1%
• error: 7.9%
• macro_f1: 91.6% Elapsed: 0:14:32 But if I run it again(as your code show, it will use the model I trained last time, which got good results), it gets a bad result as follows: => result
• total: 2,463
• correct: 876
• accuracy: 35.6%
• error: 64.4%
• macro_f1: 30.1%. I am not sure why it produces a bad result， can you give me some advice. (May it does not use the BN of the trained model)?