Hi,

This change runs the initial model step on the full context, by calling the body() function. I added an 'first' parameter defaulting to False to allow this.

I was able to install all the requirements. However while generating samples, getting the following error. I have an Intel i3 First gen Processor and running Ubuntu 18.

2019-02-16 03:12:49.453982: F tensorflow/core/platform/cpu_feature_guard.cc:37] The TensorFlow library was compiled to use AVX instructions, but these aren't available on your machine. Aborted (core dumped)

I then installed Tensorflow 1.5 (pip3 install tensorflow==1.5). The sample was generated, however another warning popped up as shown below. Will this affect the quality? Do I need to compile TensorFlow on my system?

2019-02-16 03:22:19.785441: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2

Noob here (linguist, with rudimentary knowledge of computers) I've installed the gcloud sdk but I can't get the command: sh download_model.sh 117M to run. I get: 'sh' is not recognized as an internal or external command. Any help would be greatly appreciated.

Is there a list of the flags for both conditional and unconditional models with their definitions? (I looked in the blog and paper and couldn't find any mention.)

In particular, for reproducibility purposes, it'd be great to know the definition of temperature and top_k and how choosing different values for these affect the results.

Thanks!

Hi,

I'm not familiar with gsutil. Installed it freshly using the 6 steps of : https://cloud.google.com/storage/docs/gsutil_install

Upon running the script :

When I'm not logged in on cloud.

ServiceException: 401 Anonymous caller does not have storage.objects.get access to gpt-2/models//checkpoint.
ServiceException: 401 Anonymous caller does not have storage.objects.get access to gpt-2/models//encoder.json.
ServiceException: 401 Anonymous caller does not have storage.objects.get access to gpt-2/models//hparams.json.
ServiceException: 401 Anonymous caller does not have storage.objects.get access to gpt-2/models//model.ckpt.data-00000-of-00001.
ServiceException: 401 Anonymous caller does not have storage.objects.get access to gpt-2/models//model.ckpt.index.
ServiceException: 401 Anonymous caller does not have storage.objects.get access to gpt-2/models//model.ckpt.meta.
ServiceException: 401 Anonymous caller does not have storage.objects.get access to gpt-2/models//vocab.bpe.

When I'm logged in on cloud :

AccessDeniedException: 403 [email protected] does not have storage.objects.list access to gpt-2.
AccessDeniedException: 403 [email protected] does not have storage.objects.list access to gpt-2.
AccessDeniedException: 403 [email protected] does not have storage.objects.list access to gpt-2.
AccessDeniedException: 403 [email protected] does not have storage.objects.list access to gpt-2.
AccessDeniedException: 403 [email protected] does not have storage.objects.list access to gpt-2.
AccessDeniedException: 403 [email protected] does not have storage.objects.list access to gpt-2.
AccessDeniedException: 403 [email protected] does not have storage.objects.list access to gpt-2.

Thanks

Hello,

Is it possible to predict the next word in a sentence as the research claims ? Is this locked in the bigger model ?

The python code randomnly generate sentences.

For example, like Google smart compose type "My father gifted me ", and an autocomplete prompts cheque ?

Thanks

Is it possible to release the Lambada dataset used to generate accuracy numbers in Table 3 of the paper? This would make it easier to do comparisons with other models :) @Newmu

I have removed the gsutil dependency using curl and google drive. This approach is well-known and used in several frameworks that need to download large models files (like in FastText)

If you'd like, here's a Dockerfile to toss up as an alternate installation method.

Also quickly gitignored the samples file and core file generated by running the example in the README.

Is there any way to run the Reading Comprehension: answer questions about given passages as shown in the openai example link. Can we run this using 117m model if yes than how.

Hello, what operating system do the instructions apply to? sh doesn't do anything on Windows 10. How would I install this on Win10?

Also, is the first step to clone the repo? The instructions don't seem to make sense otherwise.

Thanks.

the command : docker build --tag gpt-2 -f Dockerfile.gpu . return this :

Step 9/12 : RUN python3 download_model.py 124M
 ---> Running in afe900659249
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/certifi/core.py", line 14, in <module>
    from importlib.resources import path as get_path, read_text
ImportError: No module named 'importlib.resources'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "download_model.py", line 3, in <module>
    import requests
  File "/usr/local/lib/python3.5/dist-packages/requests/__init__.py", line 112, in <module>
    from . import utils
  File "/usr/local/lib/python3.5/dist-packages/requests/utils.py", line 24, in <module>
    from . import certs
  File "/usr/local/lib/python3.5/dist-packages/requests/certs.py", line 15, in <module>
    from certifi import where
  File "/usr/local/lib/python3.5/dist-packages/certifi/__init__.py", line 1, in <module>
    from .core import contents, where
  File "/usr/local/lib/python3.5/dist-packages/certifi/core.py", line 46, in <module>
    Resource = Union[str, "os.PathLike"]
  File "/usr/lib/python3.5/typing.py", line 552, in __getitem__
    dict(self.__dict__), parameters, _root=True)
  File "/usr/lib/python3.5/typing.py", line 512, in __new__
    for t2 in all_params - {t1} if not isinstance(t2, TypeVar)):
  File "/usr/lib/python3.5/typing.py", line 512, in <genexpr>
    for t2 in all_params - {t1} if not isinstance(t2, TypeVar)):
  File "/usr/lib/python3.5/typing.py", line 190, in __subclasscheck__
    self._eval_type(globalns, localns)
  File "/usr/lib/python3.5/typing.py", line 177, in _eval_type
    eval(self.__forward_code__, globalns, localns),
  File "<string>", line 1, in <module>
AttributeError: module 'os' has no attribute 'PathLike'
The command '/bin/sh -c python3 download_model.py 124M' returned a non-zero code: 1

I have a question about how you evaluated GPT-2 on the CoQA dataset. We are struggling to reproduce the results reported in the paper (55 F1). We evaluated gpt2-xl from HuggingFace on CoQA and got an F1 of 28.7.

We used the official dev set and evaluation script, which we downloaded from here. Although we get good answers, these answers get a lower score due to the way the original CoQA benchmark evaluator is set up. Did you evaluate it differently?

I can run the generate_unconditional_samples.py script on my GPU without issue, however, when I run the interactive_conditional_samples.py script, it crashes if there is more than one context token.

The interactive_conditional_samples.py script works fine as long as the model prompt only produces one context token, for instance using the prompt "please" produces the list of tokens [29688] and correctly generates text. However, it crashes if the model prompt produces two or more context tokens, for instance using the prompt "pig" produces the list of tokens [79, 328] and crashes immediately.

When it crashes I'm getting the error: failed to run cuBLAS routine: CUBLAS_STATUS_EXECUTION_FAILED

And a little further down I see:

Blas xGEMMBatched launch failed : a.shape=[25,2,64], b.shape=[25,2,64], m=2, n=2, k=64, batch_size=25
         [[{{node sample_sequence/model/h0/attn/MatMul}}]]
         [[sample_sequence/while/Exit_3/_1375]]

If anyone has any insight on what might be going wrong, and how I can fix it, I'd really appreciate the help.

I read the gpt2 paper, but not sure whether the pre-training data from WebText will add format information. For example, we konw data format will be english sentence = french sentencein the translation task. So during pre-training time, will we add similar promt to the training data?

Thanks!

I've tried changing temperature, top-p, presence_penalty and frequency_penalty to stop the model from repeating the same joke and other phrases with no success. I thought maybe the playground had a glitch, but this happens in the API too.

Ask for a completion on "tell me a joke" and you will likely get "why did chicken cross the road". Then variations on that "why did the duck cross the road", etc. even if the chicken tokens are removed.

The model knows other jokes.. "why don't scientists trust atoms? because they make up everything" and "why didn't the bicycle go up the hill? because it was two tired", but these only happen randomly.. once it says the chicken/road joke it cant change it (like a bad comedian ; ).

Sorry to put this here, but the GPT-3 repo is archived for new issues, but it probably also applies here.