1. Initialization -> use of the pretrained InfoNCE checkpoint.pth.tar

CUDA_VISIBLE_DEVICES=2,3 python -m torch.distributed.launch --nproc_per_node=2 main_coclr.py --net s3d --topk 5 --moco-k 2048 --dataset ucf101-2stream-2clip --seq_len 32 --ds 1 --batch_size 32 --epochs 100 --schedule 80 --name_prefix Cycle1-FlowMining_ -j 4 --pretrain /mypath/CoCLR/pretrained_by_TH/InfoNCE-ucf101-rgb-128-s3d-ep399.pth.tar /mypath/CoCLR/pretrained_by_TH/InfoNCE-ucf101-f-128-s3d-ep396.pth.tar

If i type it like this, does it Initialization? and when i do that, these words are printed out: =======Check Weights Loading====== Weights not used from pretrained file:

Weights not loaded into new model: queue queue_ptr queue_second queue_vname queue_label

Why is the weights of the pretrained model not used??

2. Alternation : In your paper, "where each cycle refers to a complete optimization of L1 and L2; meaning, the alternation only happens after the RGB or Flow network has converged."

So I entered what I wrote above into the terminal, and now I'm Training. (i.e Cycle 1 FlowMining) But acc@1 and acc@5 don't go over 1, is this the right value to have? Or is something wrong?

++ Additional If something's wrong, there's one thing I'm concerned about: in lmdb_dataset.py, I got a error for i.decode():

AttributeError: 'str' object has no attribute 'decode'

To fix this, I do that: self.db_keys_flow = msgpack.loads(txn.get(b'keys'), raw=True) self.db_order_flow = msgpack.loads(txn.get(b'order'), raw=True) . . self.db_order_rgb = msgpack.unpackb(txn.get(b'order'),raw=True) . . raw_rgb = msgpack.loads(txn.get(self.get_video_id_rgb[vname].encode('ascii')), raw=True) raw_flow = msgpack.loads(txn.get(self.get_video_id_flow[vname].encode('ascii')), raw=True)

I added "raw=True" and is this causing an error?

Hi Tengda,

I am currently trying to replicate your CoCLR result as one of the baselines in our work with the code you provide. However, I encounter some reproduction issues during the training. I understand that the code is not ready yet. It would be much appreciated if you could help us with replication. Thank you so much!

1. I found out that the Top 1 MoCo accuracy is quite low (only 4-5 percent in UCF101) with 1e-3 lr and Adam Optimizer, 1e-5 weight decay, 2048 moco queue size and 128 batch size. I wonder if you could provide a detailed training command for our reference.

2. The augmentation is not really clip-wise consistent since the value passed in is false. I wonder if this version is not final version. Could you provide the correct version of the augmentation you use?

3. Currently the code for data loader is not released and I don't know how input is prepared in data loader to be passed to TwoCropTransform and OneCropTransform. Could you please share the data loader code for our better replication?

Best Regards, Hualin

Hi Tengda, thanks for these detailed answers. I looked into all of them, seems no detailed training instruction is given on using main_classifier.py to train from scratch. The thing is, I train on UCF101 with rgb from scratch, after 500 epochs, the reported validation accuracy is 46.1%, while in test set it is only 3.41% (center crop only, top1). The detailed commands are as below:

-- Training

CUDA_VISIBLE_DEVICES=0,1 python main_classifier.py  --train_what all --epoch 500 --batch_size 24 --lr 1e-3 --wd 1e-3 --dropout 0.9 --schedule [60, 80]

-- Testing

CUDA_VISIBLE_DEVICES=0,1 python main_classifier.py --test epochxxx.pth --ten_crop

Would you like to have a quick look and help me to figure it out which configuration I made wrong? Though my computation resources is not enough, it is hard to understand why there is such a big gap between validation accuracy and test accuracy? My sincere appreciation.

Hi, I'm new to Kinetics-400 dataset. Can you provide some tutorial or instrcutions on how to generate lmdb for Kinetics-400 dataset. I find some useful message on non-local repositiry but not sure it's the proper way, thx ~

When I ran the program, I found "txn.get(self.get_video_id[name].encode('ascii')))" severely limiting the speed. Meanwhile, CPU is free. I don't know what the problem is. Hope to get help. Thanks

Hi, I am trying to download your train/val split csv file here: https://github.com/TengdaHan/CoCLR/tree/main/process_data/data/k400

But it says 403 forbidden. I believe you put it in some internal-only storage?

Hi Tengda,

Thanks for the detailed instruction for this code. I am a newbie in this field, have a very simple question regarding to table 2, and in desparate need of your help. Thanks very much in advance!

Question: From what I understand, self-supervised learning could be used to learn essencial video representation. So I guess with weights learnt by self-supervised learning methods, training the S3D network on UCF-101 will yield better results than train with random initialization. From Table 2, I suppose 90.6 is the former, and 96.8 is the latter. Would you like to explain a bit why there is such a gap?

Hi, im trying to replicate your result on the alternation stage, I now use two init models you provided (both 400~ epochs). I have two questions.

1). According to your paper, "At the alternation stage, on UCF101 the model is trained for two cycles, where each cycle includes 200 epochs, i.e. RGB and Flow networks are each trained for 100 epochs". Does that main I need to run main_coclr.py four times? each time with 100 epoch and the newest two pretrained models I have from the previous training process?

2). If so, what lr do you use in each of four 100 epochs in the alternation stage? I also checked the COCLR pretrained model you provided, it seems in 182 epoch and 109 epoch the lr is 1e-4. Is that mean I need to train the second cycles with larger lr, e.g. 1e-2, and decay down to 1e-4?

Best Regards, Yuqi

Hi Tengda, thanks for sharing your code. I found CoCLR has no data preparation instruction and could you please provide some details about data preprocessing from the raw data? I found similar instruction in DPC and MemDPC, are they feasible for CoCLR?

I have only one gpu.

I wanted to train, so I entered the terminal as follows: CUDA_VISIBLE_DEVICES=0 python -m torch.distributed.launch --nproc_per_node=1 main_coclr.py

but i got an error: subprocess.CalledProcessError: Command '['/home/junmin/anaconda3/envs/python36/bin/python', '-u', 'main_coclr.py', '--local_rank=0']' returned non-zero exit status 1.

Is there any way to train with a single GPU?

It took too long to extract the flow, so I am trying to train coclr using only rgb frames(1-stream). Is that possible?

I think it's possible when you see this part in Table 2 of your paper

If possible, can you tell me how to train using only rgb frames?

ps. Thanks for always answering in detail!

When i try ： CUDA_VISIBLE_DEVICES=0,1,2 python main_classifier.py --net s3d --dataset ucf101 --seq_len 32 --ds 1 --batch_size 32 --train_what last --epochs 30 --schedule 60 80 --optim sgd --lr 1e-1 --wd 1e-3 --final_bn --pretrain CoCLR-ucf101-rgb-128-s3d-ep182.pth Out： usage: main_classifier.py [-h] [--net NET] [--model MODEL] [--dataset DATASET] [--which_split WHICH_SPLIT] [--seq_len SEQ_LEN] [--num_seq NUM_SEQ] [--num_fc NUM_FC] [--ds DS] [--batch_size BATCH_SIZE] [--optim OPTIM] [--lr LR] [--schedule [SCHEDULE [SCHEDULE ...]]] [--wd WD] [--dropout DROPOUT] [--epochs EPOCHS] [--start_epoch START_EPOCH] [--gpu GPU] [--train_what TRAIN_WHAT] [--img_dim IMG_DIM] [--print_freq PRINT_FREQ] [--eval_freq EVAL_FREQ] [--reset_lr] [--prefix PREFIX] [-j WORKERS] [--cos] [--resume RESUME] [--pretrain PRETRAIN] [--test TEST] [--retrieval] [--dirname DIRNAME] [--center_crop] [--five_crop] [--ten_crop] main_classifier.py: error: unrecognized arguments: --final_bn

May I ask how to use the command line command --final_bn. After the above error occurred, I deleted --final_bn. Although it can run normally, it shows： Weights not loaded into new model: final_bn.weight final_bn.bias final_bn.running_mean final_bn.running_var final_bn.num_batches_tracked final_fc.0.weight final_fc.0.bias

Thanks

How can I get the two-stream fearture? And the rgb pretrained model and flow model can be use to extract two-sream feature? How can I input the command?