Dear Author,

I followed the example code using pipeline: rita_gen = pipeline('text-generation', model=model, tokenizer=tokenizer) sequences = rita_gen("MAB", max_length=20, do_sample=True, top_k=950, repetition_penalty=1.2, num_return_sequences=2, eos_token_id=2)

I got: The model 'RITAModelForCausalLM' is not supported for text-generation. Supported models are ['BartForCausalLM', 'BertLMHeadModel', 'BertGenerationDecoder', 'BigBirdForCausalLM', 'B......'

My transformers version is 4.22.0.dev0.

Please help. Thanks.

Hi, is there a way to calculate the equivalent of ESM's log_likelihood (protein stability, a.k.a fitness) for a protein sequence using this repository? Thanks in advance.

Hi,

I'm trying to run your example, but seeing

[/root/.cache/huggingface/modules/transformers_modules/lightonai/RITA_s/rita_modeling_ad8dfe2e2240d5a6abcfa9ea52f3868a35f07f4670c887c31d420f6bba0fdc5f_3db732a66a256d53b183fe3a079bbfbf2db700c63f7789178c37924666055f28.py](https://localhost:8080/#) in <module>()
     18 from transformers.utils import logging
     19 
---> 20 from .rita_configuration import RITAConfig
     21 import torch.nn.functional as F
     22 logger = logging.get_logger(__name__)

ModuleNotFoundError: No module named 'transformers_modules.lightonai.RITA_s.rita_configuration'

The attention_mask in Huggingface transformer models is padding masking, where 1 means not masked and 0 means masked. (https://huggingface.co/docs/transformers/main/en/glossary#attention-mask) I think in RITA model it is named as padding_mask.

However, RITAModelForCausalLM class does not have the argument padding_mask. Instead, it passes attention_mask argument into DecoderLayer class like this: x = layer(x, attn_mask=attention_mask), where attn_mask in RITA is the causual mask instead of padding mask. This will lead to unexpected result.

So, I think it might be better to keep the same API as other Huggingface transformer models. i.e. let users pass the Huggingface-style attention_mask into the model as padding mask.

Thank you!

Dear Author,

I am fine-tuning your pretrained RITA with a protein family data, using run_clm.py script @ huggingface. I tried this format where seq1 & seq2 are protein sequences 1 & 2 without white space:

seq1 <|endoftext|> seq2 <|endoftext|> ...

and also this format: seq1 seq2 ...

Training seemed to be successful. However, sequences generated by the fine tuned model contain a lot of '' tokens.

Please advise. Thanks.

Hi! I'm trying to run the example and have met the following errors:

>>> model = AutoModelForCausalLM.from_pretrained("lightonai/RITA_s, trust_remote_code=True")
Traceback (most recent call last):
  File "/data/home/zhongkai/miniconda/envs/protein_ssf_aws/lib/python3.8/site-packages/transformers/configuration_utils.py", line 601, in _get_config_dict
    resolved_config_file = cached_path(
  File "/data/home/zhongkai/miniconda/envs/protein_ssf_aws/lib/python3.8/site-packages/transformers/utils/hub.py", line 284, in cached_path
    output_path = get_from_cache(
  File "/data/home/zhongkai/miniconda/envs/protein_ssf_aws/lib/python3.8/site-packages/transformers/utils/hub.py", line 495, in get_from_cache
    _raise_for_status(r)
  File "/data/home/zhongkai/miniconda/envs/protein_ssf_aws/lib/python3.8/site-packages/transformers/utils/hub.py", line 417, in _raise_for_status
    raise RepositoryNotFoundError(
transformers.utils.hub.RepositoryNotFoundError: 401 Client Error: Repository not found for url: https://huggingface.co/lightonai/RITA_s,%20trust_remote_code=True/resolve/main/config.json. If the repo is private, make sure you are authenticated.

What am I doing wrong here? Appreciate the help, thanks!

Unable to run a similar script to the example

>>> from transformers import pipeline
>>> from transformers import AutoModel, AutoModelForCausalLM, AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("lightonai/RITA_s")
>>> model = AutoModelForCausalLM.from_pretrained("lightonai/RITA_s", trust_remote_code=True)
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
>>> rita_gen = pipeline('text-generation', model=model, tokenizer=tokenizer)
The model 'RITAModelForCausalLM' is not supported for text-generation. Supported models are ['XGLMForCausalLM', 'PLBartForCausalLM', 'QDQBertLMHeadModel', 'TrOCRForCausalLM', 'GPTJForCausalLM', 'RemBertForCausalLM', 'RoFormerForCausalLM', 'BigBirdPegasusForCausalLM', 'GPTNeoForCausalLM', 'BigBirdForCausalLM', 'CamembertForCausalLM', 'XLMRobertaXLForCausalLM', 'XLMRobertaForCausalLM', 'RobertaForCausalLM', 'BertLMHeadModel', 'OpenAIGPTLMHeadModel', 'GPT2LMHeadModel', 'TransfoXLLMHeadModel', 'XLNetLMHeadModel', 'XLMWithLMHeadModel', 'ElectraForCausalLM', 'CTRLLMHeadModel', 'ReformerModelWithLMHead', 'BertGenerationDecoder', 'XLMProphetNetForCausalLM', 'ProphetNetForCausalLM', 'BartForCausalLM', 'OPTForCausalLM', 'MBartForCausalLM', 'PegasusForCausalLM', 'MarianForCausalLM', 'BlenderbotForCausalLM', 'BlenderbotSmallForCausalLM', 'MegatronBertForCausalLM', 'Speech2Text2ForCausalLM', 'Data2VecTextForCausalLM'].

This is with Python3.8 and HF Transformers tokenizers-0.12.1 transformers-4.19.2

Additionally, are there more details about your prompt tuning? Curious to know how you approached it and what prompt engineering looks like for proteins as opposed to language.

Dear Author,

Thanks for releasing the RITA for protein generation! However, I wonder how can I obtain perplexity evalutation datasets used in your paper and how to calculate perplexity. Hope for your suggestions. Thanks in advance!