Hello, first of all, thank you very much for your training model, but I want to experience the process of my own training model, but the computer prompts me that it will take me a few years to train, which sounds ridiculous, but it's true. I want to ask you how long the training model has been used, and do you use GPU to train? And I have reduced the data set, but it has no effect. The training time has not been greatly reduced, or it will take several years. So I think the training time may not have much to do with the size of the data set, so what should affect the training time? It's very presumptuous to disturb you because of such a simple question, but I really need your help. Thank you very much and look forward to your reply.

Hi,

Thanks for your great work. I tried to run your generation script, but I got very disfluent "cocon_output".

For example, Given original_input_text "<|endoftext|>Once upon a time" and context_text "is perfect",

cocon_output is "<|endoftext|>Once upon a time each compuls gone 20 J-t and like to been and sa p I less millions the three is ( other remaining more we party in few V the only other end one inf only really more inf have S t and here super so Jp Lor now"

while the prependgpt2_ar_gen "is perfect<|endoftext|>Once upon a time I worked with a bike hacker who used specialised software in simple ways to automate various processes and monitor game flow. A few months ago, at the start of the second-year cycle (which sadly involved a lot of fake (and now living)"

I think the prependgpt2_ar_gen output is much better than the cocon output. I am wondering if this is correct output. Can you check to see if this is the real output from your cocon model or there is something wrong with the script? I basically follow everything in this repo.

Thanks

The pre training weight I downloaded is lost. I want to download it again, but I can't enter the website you provided again. Where else can I get the training weight? We sincerely look forward to your reply

First of all, thank you for your work and code. For the part of using classifier evaluation to generate text categories, you pointed out in the appendix that you used“ https://www.kaggle.com/rmisra/news-category-dataset ”But I can't find the text about the category of "computers" in this data. Where should I get the text data of "COMPUTERS"? This is very important to me. I look forward to your reply. Thank you again!

We meet a problem like this: Traceback (most recent call last): File "traininfer_cocon.py", line 2980, in main() File "traininfer_cocon.py", line 2783, in main global_step, tr_loss = train_cocon(args, train_dataset, model, tokenizer, cocon_block=cocon_block, disc_model=disc_model, model_config=config, transform_h_after_layernorm=args.transform_h_after_layernorm) File "traininfer_cocon.py", line 573, in train_cocon self_cocon_lm_loss_grad = torch.autograd.grad(self_cocon_lm_loss, cocon_block.cocon_attn.c_attn.weight, retain_graph=True)[0] File "/home/yingwenjing/anaconda3/envs/cocon/lib/python3.7/site-packages/torch/nn/modules/module.py", line 576, in getattr type(self).name, name)) AttributeError: 'DataParallel' object has no attribute 'cocon_attn'

We sincerely look forward to your reply.

I meet on this problem:

from .modeling_tf_utils import TFPreTrainedModel, get_initializer, keras_serializable, shape_list ImportError: cannot import name 'keras_serializable'

The version of huggingface/transformers in my env is 2.10.0.