Nice work! But I meet a strange issue when I use multi-GPUs. Specifically, the code is the same and it works well in a single GPU. But when I run it on multi-GPUS using DDP. I meet an error

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [128]] is at version 4; expected version 3 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

Does anybody meet the same issue? Hope to get your reply! thx!!

First, I have to say this is such great work, and it has been a great help in understanding the MPL code not coming from a TF background.

https://github.com/kekmodel/MPL-pytorch/blob/c66b4d3c45bc15c4be915b82eda50044f8cb3c04/main.py#L180-L182

I would not say that I am confident that this is an issue, but it is an inconsistency that I would like to bring to your attention in case it is. In the original code, they multiply the mask inside the sum as shown below, but yours multiplies outside. Does this not change the values from what they would be if multiplied inside? Or does it not matter?

If you don't do this, model's output will be random because of the dropout etc. Then your s_loss_l_new and s_loss_l_old will be random too and your dot_production won't give right feedback. Maybe this issue affects your results

I have not read the paper entirely so forgive me; but is there a way that information seeps from the unlabeled dataset to the label dataset? And if it doesn't, can we just have any dataset that could take the place of the unlabeled dataset, or does it have to look 'similar' to the label dataset? Thank you.

In here, when loading data, it seems we should shuffle the data.

def cifar10_train(params, batch_size=None):
  """Load CIFAR-10 data."""
  shuffle_size = batch_size * 16

  filenames = [os.path.join(CIFAR_PATH, 'train.bin')]
  record_bytes = 1 + (3 * 32 * 32)
  dataset = tf.data.FixedLengthRecordDataset(filenames, record_bytes)
  dataset = dataset.map(
      lambda x: _cifar10_parser(params, x, training=True),
      num_parallel_calls=tf.data.experimental.AUTOTUNE)
  dataset = dataset.shuffle(shuffle_size).repeat()
  dataset = dataset.prefetch(tf.data.experimental.AUTOTUNE)
  dataset = dataset.batch(batch_size=batch_size, drop_remainder=True)
  dataset = _optimize_dataset(dataset)

  return dataset

In your code, you didn't shuffle the dataset when loading it,

labeled_loader = DataLoader(
        labeled_dataset,
        sampler=train_sampler(labeled_dataset),
        batch_size=args.batch_size,
        num_workers=args.workers,
        drop_last=True)  # the default of `shuffle` is False

I don't really know if I am quoting the wrong TF code block of loading data, since I don't know much about TF 1.x and I know you've shuffled the labeled dataset in x_u_split(), so I just post my question here and it may not be an issue.

I think it makes sense to set shuffle of at least one of labeled_loader and unlabeled_loader to be True, so we can guarantee that we don't have epochs that have same series of batches.

Hello Sir,

I tried to train your code. But I met some error.

My start commend is as follows.

python main.py  \
        --seed 5 \
        --name cifar10-4K.5 \
        --dataset cifar10 \
        --num-classes 10 \
        --num-labeled 4000 \
        --expand-labels \
        --total-steps 300000 \
        --eval-step 1000 \
        --randaug 2 16 \
        --batch-size 128 \
        --lr 0.05 \
        --weight-decay 5e-4 \
        --dense-dropout 0.2 \
        --ema 0.995 \
        --nesterov \
        --mu 7 \
        --label-smoothing 0.15 \
        --temperature 0.7 \
        --threshold 0.6 \
        --lambda-u 8 \
        --warmup-steps 5000 \
        --uda-steps 5000 \
        --student-wait-steps 3000 \
        --amp

'''
Traceback (most recent call last):
  File "main.py", line 10, in <module>
    from torch.cuda import amp
ImportError: cannot import name 'amp' from 'torch.cuda'

How to start your code??

Thanks..

Thank you for your implementation! Here are some confusions.

In your code, I think images_uw is unlabeled and origin image, and images_us is unlabeled and RandAugment image.

In paper v.3(https://arxiv.org/pdf/2003.10580v3.pdf) Algorithm1, when "Update the student using the pseudo label", author use unlabeled and origin image to update StudentModel, but in your code,

s_images = torch.cat((images_l, images_us))
s_logits = student_model(s_images)
.....
s_loss = criterion(s_logits_us, hard_pseudo_label)

Why you use images_us (RandAugment image) as the StudentModel input to compute s_loss instead of images_uw ?

Same questions in Algorithm1 about "Compute the teacher’s feedback coefficient as in Equation 12" and "Compute the teacher’s gradient from the student’s feedback", the author use unlabeled and origin image as the input, but in your code all unlabeled image input is RandAugment, not origin. Can I ask why?

Thank you!

First of all, thank you for the great PyTorch implementation.

But I can't get how the optimization procedure of the teacher network (eq.3 of the paper) is implemented. The code should calculate the 2nd-order derivative during training, which is missing in the current version of the code. Would you check the code again and let me know whether there is something I'm missing?

Thank you!

Anyone meet this problem??? I could not deal with it. Detail: → TypeError: init() got an unexpected keyword argument 'label_smoothing'

in main.py line 546 → criterion = create_loss_fn(args) and in utils.py line 32 → criterion = nn.CrossEntropyLoss(label_smoothing=args.label_smoothing)

Thanks!

Hello, Thank you your code. I have some problem like :

Traceback (most recent call last): File "C:\Users\user\Desktop\Pyler\MPL-pytorch\main.py", line 156, in train_loop (images_uw, images_us), _ = unlabeled_iter.next() File "C:\Users\user\anaconda3\envs\pyler\lib\site-packages\torch\utils\data\dataloader.py", line 517, in next data = self._next_data() File "C:\Users\user\anaconda3\envs\pyler\lib\site-packages\torch\utils\data\dataloader.py", line 1199, in _next_data return self._process_data(data) File "C:\Users\user\anaconda3\envs\pyler\lib\site-packages\torch\utils\data\dataloader.py", line 1225, in _process_data data.reraise() File "C:\Users\user\anaconda3\envs\pyler\lib\site-packages\torch_utils.py", line 429, in reraise raise self.exc_type(msg) ValueError: Caught ValueError in DataLoader worker process 0. Original Traceback (most recent call last): File "C:\Users\user\anaconda3\envs\pyler\lib\site-packages\torch\utils\data_utils\worker.py", line 202, in _worker_loop data = fetcher.fetch(index) File "C:\Users\user\anaconda3\envs\pyler\lib\site-packages\torch\utils\data_utils\fetch.py", line 44, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "C:\Users\user\anaconda3\envs\pyler\lib\site-packages\torch\utils\data_utils\fetch.py", line 44, in data = [self.dataset[idx] for idx in possibly_batched_index] File "C:\Users\user\Desktop\Pyler\MPL-pytorch\data.py", line 180, in getitem img = self.transform(img) File "C:\Users\user\Desktop\Pyler\MPL-pytorch\data.py", line 159, in call aug = self.aug(x) File "C:\Users\user\anaconda3\envs\pyler\lib\site-packages\torchvision\transforms\transforms.py", line 60, in call img = t(img) File "C:\Users\user\Desktop\Pyler\MPL-pytorch\augmentation.py", line 211, in call img = op(img, v=self.m, max_v=max_v, bias=bias) File "C:\Users\user\Desktop\Pyler\MPL-pytorch\augmentation.py", line 165, in TranslateYConst return img.transform(img.size, PIL.Image.AFFINE, (1, 0, 0, 0, 1, v), RESAMPLE_MODE) File "C:\Users\user\anaconda3\envs\pyler\lib\site-packages\PIL\Image.py", line 2490, in transform im.__transformer( File "C:\Users\user\anaconda3\envs\pyler\lib\site-packages\PIL\Image.py", line 2556, in __transformer raise ValueError( ValueError: Unknown resampling filter (None). Use Image.NEAREST (0), Image.BILINEAR (2) or Image.BICUBIC (3)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users\user\Desktop\Pyler\MPL-pytorch\main.py", line 633, in main() File "C:\Users\user\Desktop\Pyler\MPL-pytorch\main.py", line 626, in main train_loop(args, labeled_loader, unlabeled_loader, test_loader, File "C:\Users\user\Desktop\Pyler\MPL-pytorch\main.py", line 162, in train_loop (images_uw, images_us), _ = unlabeled_iter.next() File "C:\Users\user\anaconda3\envs\pyler\lib\site-packages\torch\utils\data\dataloader.py", line 517, in next data = self._next_data() File "C:\Users\user\anaconda3\envs\pyler\lib\site-packages\torch\utils\data\dataloader.py", line 1199, in _next_data return self._process_data(data) File "C:\Users\user\anaconda3\envs\pyler\lib\site-packages\torch\utils\data\dataloader.py", line 1225, in _process_data data.reraise() File "C:\Users\user\anaconda3\envs\pyler\lib\site-packages\torch_utils.py", line 429, in reraise raise self.exc_type(msg) ValueError: Caught ValueError in DataLoader worker process 0. Original Traceback (most recent call last): File "C:\Users\user\anaconda3\envs\pyler\lib\site-packages\torch\utils\data_utils\worker.py", line 202, in _worker_loop data = fetcher.fetch(index) File "C:\Users\user\anaconda3\envs\pyler\lib\site-packages\torch\utils\data_utils\fetch.py", line 44, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "C:\Users\user\anaconda3\envs\pyler\lib\site-packages\torch\utils\data_utils\fetch.py", line 44, in data = [self.dataset[idx] for idx in possibly_batched_index] File "C:\Users\user\Desktop\Pyler\MPL-pytorch\data.py", line 180, in getitem img = self.transform(img) File "C:\Users\user\Desktop\Pyler\MPL-pytorch\data.py", line 159, in call aug = self.aug(x) File "C:\Users\user\anaconda3\envs\pyler\lib\site-packages\torchvision\transforms\transforms.py", line 60, in call img = t(img) File "C:\Users\user\Desktop\Pyler\MPL-pytorch\augmentation.py", line 211, in call img = op(img, v=self.m, max_v=max_v, bias=bias) File "C:\Users\user\Desktop\Pyler\MPL-pytorch\augmentation.py", line 111, in ShearX return img.transform(img.size, PIL.Image.AFFINE, (1, v, 0, 0, 1, 0), RESAMPLE_MODE) File "C:\Users\user\anaconda3\envs\pyler\lib\site-packages\PIL\Image.py", line 2490, in transform im.__transformer( File "C:\Users\user\anaconda3\envs\pyler\lib\site-packages\PIL\Image.py", line 2556, in __transformer raise ValueError( ValueError: Unknown resampling filter (None). Use Image.NEAREST (0), Image.BILINEAR (2) or Image.BICUBIC (3)

Something wrong happend. I think it is because of pytorch version. My Pytorch version is 1.8.0. How can I fix it?

Thank you

In the link, I see that the learning rate (of student?) starts from 0.5, but if I understand your code correctly, the learning rate should be 0 during the first 3000 steps as when training CIFAR10 model, you specified the num_wait_step to be 3000. Also, you specified the num_warmup_steps to be 5000. Can you please explain a bit? Thanks!

It is a very good job!

But I am confused about the derivation in Equation 10 of the appendix. How can we apply the REINFORCE equation to achieve Equation 10?

Hi, thanks for your great code, I am confused about accuracy with or without finetune on CIFAR-10-4K. Results show that accuracy with and without finetune on CIFAR-10-4K are 96.01% and 96.08%, which means there is no significant increase of accuracy with finetune. And I wonder is it means we could use only unlabeled data (with pseudo label generated by teacher model) to train our student model and obtain accuracy almost equal to train it with labeled data? So it indicates indirectly that pseudo labels generated by teacher model are high-quality? I would appreciated if you could give me a response. Thanks!

https://github.com/kekmodel/MPL-pytorch/blob/7fb5b40cd53179bf4c09ef0f916815c3272d3e9d/main.py#L197

At this point hard_pseudo_label is a batch size by 1 array of integer indexes, not a one hot encoding.

Is this correct when calculating cross entropy?

https://github.com/kekmodel/MPL-pytorch/blob/7fb5b40cd53179bf4c09ef0f916815c3272d3e9d/data.py#L138

Hey all,

I've been working my way through this implementation and cannot work out why the expand labels options exists? It seems that even if the labels aren't expanded the data loader will loop anyway.

Can anyone explain why this is needed?