It seems that some new version of fast ai broke this repository. I'm getting an error

conda ImportError: cannot import name progress_bar from fastprogress

which disappears if I install fastai==1.0.59.

But there are still errors with different packages like transformers. Could you please send the output of your pip freeze inside the working environment?

Please provide the working ipynb, i'm trying to run it in google colab (using transformers), but it's so messy it won't even run and where's the transformers compliant tokenizer?

upd 17.07: no response

Hello,

sorry if this is a silly question. I am trying to finetune the English GPT-2 model in my language. I tried without the "while loop" command and I got a memory issue problem (oom error) and the algorithm crashed. So then I tried to run it like this:

export TRAIN_FILE=/ru_transformers/fulljan21
export CUDA_VISIBLE_DEVICES=2
export MODEL_SIZE=gpt2-large
export OUTPUT=output_yt/lfeb
export BS=1
export LR=1e-5

  while true
  do
      python run_lm_finetuning.py \
                --output_dir=$OUTPUT \
          --model_type=gpt2 \
          --model_name_or_path=$OUTPUT \
          --do_train \
          --train_data_file=$TRAIN_FILE \
          --per_gpu_train_batch_size $BS \
          --save_steps=10000 \
          --logging_steps=10 \
          --fp16 \
          --fp16_opt_level O2 \
          --warmup_samples 16000 \
          --learning_rate $LR \
          --overwrite_output_dir \
          --tokenizer_class YTEncoder \
          --tokenizer_name bpe/yt.model \
          --do_eval \
          --evaluate_during_training \
          --eval_steps 1000 \
          --eval_data_file=./data/classic/valid \
          --save_total_limit 30 \
          --num_train_epochs 10.0 \
          --unfreeze_level 0

      sleep 1
  done

So now the model is running for around two weeks and it keeps evaluating. I cannot see the percentage of epochs or a sample.txt file that I could see when I was running the model with a very small sample of data (2gb). Now my files are around 40gb. I am using a NVIDIA Corporation GV102 graphic card and she is working in aroung 40%. Am I doing something wrong?

Currently I have this output:

 02/10/2021 14:35:22 - INFO - __main__ -   Loading features from ./evaluate_dir
 ./evaluate_dir

1675459███████████████████████████████████████████████████████████████████████████████████████████████| 100.00% [2259/2259 00:17<00:00]

 02/10/2021 14:35:39 - INFO - __main__ -   ***** Running evaluation checkpoint-27000 *****
 02/10/2021 14:35:39 - INFO - __main__ -     Num examples = 1675459
 02/10/2021 14:35:39 - INFO - __main__ -     Batch size = 4
 Evaluating:  17%|███████████████████████████████▏                                                                                                                                                   | 72983/418865 [58:22<4:35:44, 20.91it/s]

Thank you

Hello! Your work is amazing! Thank you. Can you provide some instruction on how I can fine tune your models on some specific corpus? How long such process would be?

Hi! Thank your for sharing your work! After installation i try to run rest.py and get this error:

ALL_MODELS = sum((tuple(conf.pretrained_config_archive_map.keys()) for conf in (GPT2Config, OpenAIGPTConfig, XLNetConfig, TransfoXLConfig)), ())
AttributeError: type object 'GPT2Config' has no attribute 'pretrained_config_archive_map'

Hello and thank you for the great work.

This might be a rather stupid question, but I'm just beginning with NPL. Apologies in advance. Could you give me a brief intro with regards to the files generated in the models ?

I am asking this because when I try to load the head model class and the tokenizer class I get the following error: We assumed '../russian_models/gpt2/m_checkpoint-3364613' was a path or url to a directory containing vocabulary files named ['vocab.json', 'merges.txt'] but couldn't find such vocabulary files at this path or url

Of course, those files are not present there, but I'm not sure where to start at the moment. And a second question if possible, are you licensing this project under MIT by any chance?

Thank you in advance.

Hi,

I'm using your run_lm_finetuning.py script. It works but I would like to know why in the starting of the training, I get:

• 3 times the following sentence: Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to ...
• and the following warning:

/opt/anaconda3/envs/gpt/lib/python3.7/site-packages/torch/optim/lr_scheduler.py:91: 
UserWarning: Seems like `optimizer.step()` has been overridden after learning rate scheduler 
initialization. Please, make sure to call `optimizer.step()` before `lr_scheduler.step()`. 
See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
  "https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)

It is strange as we can see that optimizer.step() is well called before scheduler.step() from line 359.

Any thought about that?

When I try to continue my story, this happens over and over again on different devices, which started yesterday in the evening. The neural network doesn't respond on any device, and many people report about having the same issue.

Hello,

I have 28GB of text and I want to train from scratch. I have 4 GPUs (product: GV102 vendor: NVIDIA Corporation) and the algorithm crashes due to memory issues. I saw in your readme this

 # My dataset is 230Gb and it doesn't fit in RAM, so each epoch is a random sample from it. That is why the loop.
while true
do
python run_lm_finetuning.py \
    --output_dir=$OUTPUT \
    --model_type=gpt2 \
    --model_name_or_path=$OUTPUT \
    --do_train \
    --train_data_file=$TRAIN_FILE \
    --per_gpu_train_batch_size $BS \
    --save_steps=10000 \
    --logging_steps=10 \
    --fp16 \
    --fp16_opt_level O2 \
    --warmup_samples 16000 \
    --learning_rate $LR \
    --overwrite_output_dir \
    --tokenizer_class YTEncoder \
    --tokenizer_name bpe/yt.model \
    --do_eval \
    --evaluate_during_training \
    --eval_steps 1000 \
    --eval_data_file=./data/classic/valid \
    --save_total_limit 30 \
    --num_train_epochs 10.0 \
    --unfreeze_level 0

sleep 1

done

So If I use these parameters to train the model when will it stop, since you say while true. Do you believe this will fix the memory problem? Also it uses eventually all the training data or it is random as you say?

Thank you

I was trying to get the TPU training run on Google Colab TPU. There is this TPU MNIST Demo that works fine with xla. (https://colab.sandbox.google.com/github/pytorch/xla/blob/master/contrib/colab/mnist-training-xrt-1-15.ipynb) so I thought I should be able to run your test_train_mp_mnist.py. But running into issues. Even the tpu_lm_finetuning.py failed to run with some thread join error. Do you have any plans to make your code run on Google Colab TPU? That would be very helpful.

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch_xla/distributed/xla_multiprocessing.py", line 119, in _start_fn
    fn(gindex, *args)
  File "/content/ru_transformers/tpu/test_train_mp_mnist.py", line 180, in _mp_fn
    accuracy = train_mnist()
  File "/content/ru_transformers/tpu/test_train_mp_mnist.py", line 79, in train_mnist
    transforms.Normalize((0.1307,), (0.3081,))]))
  File "/usr/local/lib/python3.6/dist-packages/torchvision/datasets/mnist.py", line 71, in __init__
    self.download()
  File "/usr/local/lib/python3.6/dist-packages/torchvision/datasets/mnist.py", line 144, in download
    read_image_file(os.path.join(self.raw_folder, 'train-images-idx3-ubyte')),
  File "/usr/local/lib/python3.6/dist-packages/torchvision/datasets/mnist.py", line 483, in read_image_file
    x = read_sn3_pascalvincent_tensor(f, strict=False)
  File "/usr/local/lib/python3.6/dist-packages/torchvision/datasets/mnist.py", line 461, in read_sn3_pascalvincent_tensor
    magic = get_int(data[0:4])
  File "/usr/local/lib/python3.6/dist-packages/torchvision/datasets/mnist.py", line 426, in get_int
    return int(codecs.encode(b, 'hex'), 16)
ValueError: invalid literal for int() with base 16: b''

Good day guys, I added your system to my VK bot. But I didn't see limits. I hope my small bot won't be a problem for you. Thanks!

My commit is: https://github.com/Disinterpreter/perl-vk-bot/commit/1f0fdf80c9ab264befffc6052c15a2e2f423bd29#diff-b8426a177e7f1a5b01a99d88a34d1c93