Thanks for your interesting work! I am trying to do some experiments using your code. I find that it is too expensive to augment unlabled data through back translation by myself because of limited resource. I can only find back translation data of Yahoo dataset in this codebase. Can you provide all back translation data? Thanks!

Hi @diyiy @jiaaoc ,

just a quick question: Can I also use the code found in this repo to reproduce your UDA results reported in the paper? If so, how?

Thanks :)

Hey,

I was trying to test your model for my custom data which contains both labelled and unlabelled sentences. Though I am not sure how I need to format, structure and keep my unlabelled data.

I was able to run it successfully on the preprocessed Yahoo dataset but there were only labelled examples present in it as far as I could observe.

Sorry for disturbing you again. I tried to change the max_seq_len to 512, and change the BERT to Electra. but I'm not sure that 102 in this line of code needs to be changed

https://github.com/GT-SALT/MixText/blob/f17198d98e1bbb012d8e33cd26e228dd43bb4673/code/train.py#L313

Hello, I am following your work! I have a question about the figure 2 in your paper. Why the figure shows that you only augment the unlabel data ? Looking forward your answer!

Hi, Firstly I quite liked the paper and enjoyed reading it. I was trying to implement it and tried using this code for implementation on yahoo answers dataset but unfortunately the best accuracy does not cross 0.24. Since I have 2 gpus at my disposal I made the batch-size 2 and batch-size u 4 (also tried batch size 3 and batch size-u 4) and val-iteration 2000. So I was wondering, if you could let me know should I change other parameters to make it work? ( I understand that reducing the batch size should have some impact but not sure that the impact can be this significant so thought it might need to do with some other factors or weighting)?

The command I used (i downloaded the dataset from the link provided and placed it in the directory)

python ./code/train.py --gpu 0,1 --n-labeled 10 --data-path ./data/yahoo_answers_csv/ --batch-size 2 --batch-size-u 4 --epochs 20 --val-iteration 2000 --lambda-u 1 --T 0.5 --alpha 16 --mix layers-set 7 9 12 --lrmain 0.000005 --lrlast 0.0005

The second question I had was the use of args.val_iteration? As I understand it is in a way number of batches to be processed in an epoch. So I was wondering how does it work if my number of labeled data per class is 10 == 100 labeled examples(for 10 classes in yahoo answers). So if I have batch size of 2 that would be 50 batches for data loader? So does it repeat instances in cyclic manner or just randomly pics 2 instances always?

Thanks

I was trying to use the backtranslation code with the IMDB polarity dataset. But the sequences are very long and fairseq is giving an error. How you handle such long sequences? Thanks for the code.

I've been working on this and got results close to what was mentioned in the paper for the IMDB dataset. However, i'm facing a couple of issues.

1. Changing the sequence length that BERT accepts hasn't been straightforward. I want it changed to 128. I couldn't see that in 'mixtext.py'. 'normal_bert.py' contains 'length=256' in the definition for 'forward' but I'm unsure if this the one to change as I don't see 'length' being used anywhere else. Please let me know how to change the sequence length for both normal_bert and mix_text.

2. I tried to run this on the sklearn's NewsGroup dataset - which contains 20 labels. I've appropriately changed the 'read_data.py' to take in this data & also performed back-translations as mentioned. But I couldn't get good results with 'MixText'. Whatever the amount of labeled data that I use, I get almost similar, very low accuracies. Training time also remains the same irrespective of what the labeled data is. I'm assuming there is no issue with my data reading & pre-processing because 'normal_bert' runs fine. Please do let me know if there is any known issue with using MixText for a multi-class dataset.

Thanks!

Hello, what version of transformers are you using in the code? Tried to run and instance the MixText model but got an error of no attribute 'chunk_size_feed_forward' when creating the layers.

model = MixText(n_labels, args['mix_option']).cuda()

And using 'bert-base-uncased' as pretrained model.

I have a question about what exactly is the supervised loss mentioned in paper or here in the code this Lx. It is not exactly cross entropy so was a bit confused, a bit more explanation about this loss function would be really helpful?

Lx = - torch.mean(torch.sum(F.log_softmax(outputs_x, dim=1) * targets_x, dim=1))

Hi,

I'm currently working on your repository, and it seems that some special tokens for the BERT are missing. (e.g. or )

From my understanding, those special characters are not concatenated to the input text and are fed to the BERT model.

Is it right? Or am I missing it?

Thank you

Hello: I am trying to run this project, but the linked URL of The pre-processed Yahoo Answer dataset is invalid. Could you please make up the URL of The pre-processed Yahoo Answer dataset?

Thanks, Rosit

Hello!

I am trying to reproduce your work, but I have issues with the dependencies. Can you provide the yml file of the conda environment that you used or at least specify the version of each package (transformers torch, ...)?

Thanks, Taha

%run /code/train.py --gpu=0 --n-labeled=10 --data-path /yahoo_answers_csv/ --batch-size=4 --batch-size-u=8 --epochs=50 --val-iteration=20 --lambda-u=0 --T=0.5 --alpha=16 --mix-layers-set 7 9 12 --separate-mix=True

train(labeled_trainloader, unlabeled_trainloader, model, optimizer, scheduler, criterion, epoch, n_labels, train_aug) 204 (inputs_u, inputs_u2, inputs_ori), (length_u, --> 205 length_u2, length_ori) = unlabeled_train_iter.next() 206 except:

/torch/utils/data/dataloader.py in next(self) 520 self._reset() --> 521 data = self._next_data() 522 self._num_yielded += 1

/torch/utils/data/dataloader.py in _next_data(self) 560 index = self._next_index() # may raise StopIteration --> 561 data = self._dataset_fetcher.fetch(index) # may raise StopIteration 562 if self._pin_memory:

/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index) 43 if self.auto_collation: ---> 44 data = [self.dataset[idx] for idx in possibly_batched_index] 45 else:

/torch/utils/data/_utils/fetch.py in (.0) 43 if self.auto_collation: ---> 44 data = [self.dataset[idx] for idx in possibly_batched_index] 45 else:

/code/read_data.py in getitem(self, idx) 209 if self.aug is not None: --> 210 u, v, ori = self.aug(self.text[idx], self.ids[idx]) 211 encode_result_u, length_u = self.get_tokenized(u)

/code/read_data.py in call(self, ori, idx) 22 def call(self, ori, idx): ---> 23 out1 = self.de[idx] 24 out2 = self.ru[idx]

KeyError: 9226

May I ask what is the reason for this place? Thank you very much.

I recently studied your project and carefully read your paper on ACL. However, when I tried to run your code, I didn't find the setting of the parameter epoch, and the setting of other parameters in the code were vague, which was not fully specified in the paper, especially the parameter epoch. Therefore, I can't get the results in your paper. Can you provide me with a table on the parameter setting of each dataset?

We look forward to your reply