Hi,

thank you for your great contribution to this interesting template NER topic. I wonder if it's possible to adapt this code to another language. I've included the model and tokenizer in the MODEL_CLASSES (and other parts) since it has a different tokenizer compared to English BART.

MODEL_CLASSES = {
    "auto": (AutoConfig, AutoModel, AutoTokenizer),
    "bart": (BartConfig, BartForConditionalGeneration, BartTokenizer),
    "bert": (BertConfig, BertModel, BertTokenizer),
    "roberta": (RobertaConfig, RobertaModel, RobertaTokenizer),
    "indobart": (MBartConfig, MBartForConditionalGeneration, IndoNLGTokenizer)
}

Could you share some hints on which part I should put attention to when adding other pre-trained models/language to the code?

Thank you so much for your help!

Best, Oryza

Hi, Thank you for your nice work. Could you please release the test data of this code? I have no idea what format of the data should be applied in inference.py.

Hi, Seq2SeqModel.predict function predicts one entity at a time. e.g.

predict("Tesla, IBM, and Amazon are the good tech companies") -> "Tesla is an Organization" What about extracting IBM and Amazon as well at the same time!

The acc calculated in the project only computed the absolute accuracy of generated sequence, which contained the tokens like: "is," "a," "an," "entity."

I calculated the P, R, F1 of the entity token content and entity class of the generated sequence（train based on CoNLL03）. The evaluation results are inconsistent with the paper, with the ' organization' entity obtaining only 0.58 F1. Could you please publish the dataset of the paper and the complete evaluation methods?

Hi, Thank you for sharing. Could you please provide the version of transformers and other packages? I encountered the problem when running the inference.py (Line: output = model(input_ids=input_ids.to(device), decoder_input_ids=output_ids[:, :output_ids.shape[1] - 2].to(device))[0). The error is as below:

Hi, I tried your code and once I use your demo to generate output I expected ["original text","predict entity with prompt"], however it just output ['JapanJapanJapan JapanJapanJapan']. Is that any layer you add to fine-tuning but not shown in code?

The csv file in the data directory seems only use the gold entity as the answer sentence. Shall I add some negative samles like "by is not a named entity" as an answer sentence to the csv file for training? Thanks.

Here -

for i in range(len(input_TXT_list)):
     words = []
     # for j in range(1, min(9, len(input_TXT_list) - i + 1)):
     for j in range(1, 9):
         word = (' ').join(input_TXT_list[i:i+j])
         words.append(word)
     print(words)

This is the output on calling the prediction function with previous version -

prediction("I made ghee from butter chilli")

['I', 'made', 'ghee', 'from', 'butter', 'chilli']
['I', 'I made', 'I made ghee', 'I made ghee from', 'I made ghee from butter', 'I made ghee from butter chilli', 'I made ghee from butter chilli', 'I made ghee from butter chilli']
['made', 'made ghee', 'made ghee from', 'made ghee from butter', 'made ghee from butter chilli', 'made ghee from butter chilli', 'made ghee from butter chilli', 'made ghee from butter chilli']
['ghee', 'ghee from', 'ghee from butter', 'ghee from butter chilli', 'ghee from butter chilli', 'ghee from butter chilli', 'ghee from butter chilli', 'ghee from butter chilli']
['from', 'from butter', 'from butter chilli', 'from butter chilli', 'from butter chilli', 'from butter chilli', 'from butter chilli', 'from butter chilli']
['butter', 'butter chilli', 'butter chilli', 'butter chilli', 'butter chilli', 'butter chilli', 'butter chilli', 'butter chilli']
['chilli', 'chilli', 'chilli', 'chilli', 'chilli', 'chilli', 'chilli', 'chilli']

After this fix, the output is -

['I', 'made', 'ghee', 'from', 'butter', 'chilli']
['I', 'I made', 'I made ghee', 'I made ghee from', 'I made ghee from butter', 'I made ghee from butter chilli']
['made', 'made ghee', 'made ghee from', 'made ghee from butter', 'made ghee from butter chilli']
['ghee', 'ghee from', 'ghee from butter', 'ghee from butter chilli']
['from', 'from butter', 'from butter chilli']
['butter', 'butter chilli']
['chilli']

Multiple occurrences of the same string happen when j > (len(input_TXT_list) - i), increasing computation time. This commit fixes this.

I tried to run inference.py but it gives the error

/configuration_utils.py", line 609, in _get_config_dict
    user_agent=user_agent,
  File "/usr/local/lib/python3.7/dist-packages/transformers/utils/hub.py", line 292, in cached_path
    local_files_only=local_files_only,
  File "/usr/local/lib/python3.7/dist-packages/transformers/utils/hub.py", line 495, in get_from_cache
    _raise_for_status(r)
  File "/usr/local/lib/python3.7/dist-packages/transformers/utils/hub.py", line 418, in _raise_for_status
    f"401 Client Error: Repository not found for url: {response.url}. "
transformers.utils.hub.RepositoryNotFoundError: 401 Client Error: Repository not found for url: https://huggingface.co/checkpoint-3060/resolve/main/config.json. If the repo is private, make sure you are authenticated.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "inference.py", line 104, in <module>
    model = BartForConditionalGeneration.from_pretrained('./checkpoint-3060')
  File "/usr/local/lib/python3.7/dist-packages/transformers/modeling_utils.py", line 1934, in from_pretrained
    **kwargs,
  File "/usr/local/lib/python3.7/dist-packages/transformers/configuration_utils.py", line 526, in from_pretrained
    config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/configuration_utils.py", line 553, in get_config_dict
    config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/configuration_utils.py", line 614, in _get_config_dict
    f"{pretrained_model_name_or_path} is not a local folder and is not a valid model identifier listed on "
OSError: ./checkpoint-3060 is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo with `use_auth_token` or log in with `huggingface-cli login` and pass ``use_auth_token=True

Hi,

would you mind explaining some hard-coded numbers in the template_entity function from inference.py?

def template_entity(words, input_TXT, start):
    # input text -> template
    words_length = len(words)
    words_length_list = [len(i) for i in words]
    input_TXT = [input_TXT]*(5*words_length)

    input_ids = tokenizer(input_TXT, return_tensors='pt')['input_ids']
    model.to(device)
    template_list = [" is a location entity .", " is a person entity .", " is an organization entity .",
                     " is an other entity .", " is not a named entity ."]
    entity_dict = {0: 'LOC', 1: 'PER', 2: 'ORG', 3: 'MISC', 4: 'O'}
    temp_list = []
    for i in range(words_length):
        for j in range(len(template_list)):
            temp_list.append(words[i]+template_list[j])

    output_ids = tokenizer(temp_list, return_tensors='pt', padding=True, truncation=True)['input_ids']
    output_ids[:, 0] = 2
    output_length_list = [0]*5*words_length


    for i in range(len(temp_list)//5):
        base_length = ((tokenizer(temp_list[i * 5], return_tensors='pt', padding=True, truncation=True)['input_ids']).shape)[1] - 4
        output_length_list[i*5:i*5+ 5] = [base_length]*5
        output_length_list[i*5+4] += 1

    score = [1]*5*words_length
    with torch.no_grad():
        output = model(input_ids=input_ids.to(device), decoder_input_ids=output_ids[:, :output_ids.shape[1] - 2].to(device))[0]
        for i in range(output_ids.shape[1] - 3):
            # print(input_ids.shape)
            logits = output[:, i, :]
            logits = logits.softmax(dim=1)
            # values, predictions = logits.topk(1,dim = 1)
            logits = logits.to('cpu').numpy()
            # print(output_ids[:, i+1].item())
            for j in range(0, 5*words_length):
                if i < output_length_list[j]:
                    score[j] = score[j] * logits[j][int(output_ids[j][i + 1])]

    end = start+(score.index(max(score))//5)
        # score_list.append(score)
    return [start, end, entity_dict[(score.index(max(score))%5)], max(score)] #[start_index,end_index,label,score]

I learned from the opened issues that the 5s are the length of the template_list but how about the other numbers?

It would be a great help if you could response to this, thank you in advance!

When I train the model with custom labels, the training code works well. However, Adapting Inference.py code to my custom trained model does not work.

I change the Inference.ipynb code to adapt my 11 labels as follows:

    LABELS=["Adjective","API","Core","GUI","Hardware","Language","Platform","Standard","User","Verb","O"]
    template_list=[" is a %s entity"%(e) for e in LABELS]
    entity_dict={i:e for i, e in enumerate(LABELS)}

Here is loading checkpoint

tokenizer = BartTokenizer.from_pretrained('facebook/bart-large')
model = BartForConditionalGeneration.from_pretrained('./outputs/best_model')

Here is the inference and the error

prediction("As a user I should be able to use the attribute type User in my queries.")

RuntimeError
----> 2 prediction("As a user I should be able to use the attribute type User in my queries.") /usr/local/lib/python3.7/dist-packages/transformers/models/bart/modeling_bart.py in _shape(self, tensor, seq_len, bsz) 157 def _shape(self, tensor: torch.Tensor, seq_len: int, bsz: int): --> 158 return tensor.view(bsz, seq_len, self.num_heads, self.head_dim).transpose(1, 2).contiguous() RuntimeError: shape '[88, -1, 16, 64]' is invalid for input of size 778240