How do I make use of this FewNERD model to do inferencing on my data-set after training on FewNERD data. The idea is to see how it performs on my custom -data?

Step1: I train the proto model using FEWNerd data as mentioned:

python3 train_demo.py --mode inter --lr 1e-4 --batch_size 8 --trainN 5 --N 5 --K 1 --Q 1 --train_iter 10000 --val_iter 500 --test_iter 5000 --val_step 1000 --max_length 64 --model proto --tau 0.32 Once training is complete:

I have data-set which is in this format and has entities as ['dispute_amount', 'dispute_date', 'competitors]

My test-data is in this format?

{"
word": 
          [ 
              ["Dispute", "Case", "ID", "MM-Z-*********", "the", "amount", "of", "$99.99", "should", "be", "AU", "dollars", "total", "is", "$86.85", "US", "dollars."], 
               ["8:27am,", "I", "started", "a", "claim", "for", "which", "I", "was", "refunded", "for", "one", "item,", "but", "not", "for", "the", "other,", "from", "the", "same", "seller."] 
          ], 

"label": [ 

                 ["O", "O", "O", "O", "O", "O", "O", "dispute_amount", "O", "O", "O", "O", "O", "O", "dispute_amount", "dispute_amount", "dispute_amount"], 
                 ["dispute_date", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O"] 
              ],  

"types": [["dispute_amount"], ["dispute_date"]]

}

How can I test and print outputs on my test data?

@cyl628 @ningding97

Hello, thanks for this project. I was able to correctly train a structshot using the train script. Could you show how to correctly run the inference for an input sequence? In my understanding, the loading would look like

import os
import torch

from fewnerd.util.word_encoder import BERTWordEncoder
from fewnerd.model.proto import Proto
from fewnerd.model.nnshot import NNShot

# cache dir
cache_dir = os.getenv("cache_dir", "../../models")
model_path = 'structshot-inter-5-5-seed0.pth.tar'
model_name = 'structshot'
pretrain_ckpt = 'bert-base-uncased'
max_length = 100

# BERT word encoder
word_encoder = BERTWordEncoder(
        pretrain_ckpt,
        max_length)

if model_name == 'proto':
    # use dot instead of L2 distance for proto
    model = Proto(word_encoder, dot=True)
elif model_name == 'nnshot':
    model = NNShot(word_encoder, dot=False)
elif model_name == 'structshot':
    model = NNShot(word_encoder, dot=False)

model.load_state_dict(torch.load(os.path.join(cache_dir, model_path)))

but I get some errors on the state dicts (RuntimeError: Error(s) in loading state_dict for NNShot:) in this way...

Thank you in advance!

Inside data/episode-data/inter/train_5_5.jsonl I see lot of training, test and dev data. They are like support and query .

In the data/inter/train.txt I see a different form of training data without support and query .

What is the purpose to keep these 2 formats of data?

Does both are used to train few-shot learning?

@cyl628

Inside data/episode-data/inter/ I see lot of training, test and dev data. I may be asking few silly questions , please pardon.

I was exploring train_5_5.jsonl. What does train_5_5.jsonl signifies? Does it has to do anything with support and query set?

Here is one example:

I see support has 14 sentences and Query has 15 sentences..

So, this one example mentioned below : support and query is fed into model as single example? Support is used to train the model? Then why Query is used?? I am seeing this training data structure for the first time. Could you give me more insights in lay man terms how training happens?

{
  "support":
          {"word":
                  [
                    ["averostra", ",", "or", "``", "bird", "snouts", "''", ",", "is", "a", "clade", "that", "includes", "most", "theropod", "dinosaurs", "that", "have", "a", "promaxillary", "fenestra", "(", "``", "fenestra", "promaxillaris", "``", ")", ",", "an", "extra", "opening", "in", "the", "front", "outer", "side", "of", "the", "maxilla", ",", "the", "bone", "that", "makes", "up", "the", "upper", "jaw", "."],
                    ["since", "that", "time", ",", "the", "squadron", "made", "several", "extended", "indian", "ocean", ",", "mediterranean", "sea", ",", "and", "north", "atlantic", "deployments", "as", "part", "of", "cvw-1", "/", "cv-66", ",", "until", "the", "decommissioning", "of", "uss", "``", "america", "''", "in", "1996", "."],
                    ["the", "alpha-gal", "allergy", "is", "believed", "to", "result", "from", "tick", "bites", "."],
                    ["interaction", "was", "shown", "to", "occur", "with", "the", "dna", "-directed", "rna", "polymerase", "ii", "subunit", ",", "rpb1", ",", "of", "rna", "polymerase", "ii", "during", "both", "mitosis", "and", "interphase", "."],
                    ["he", "is", "also", "responsible", "for", "programming", "on", "diablo", "ii", ",", "the", "development", "of", "the", "battle.net", "game", "server", "network", ",", "and", "the", "quake", "2", "mod", "loki", "'s", "minions", "capture", "the", "flag", "."],
                    ["minix", "was", "first", "released", "in", "1987", ",", "with", "its", "complete", "source", "code", "made", "available", "to", "universities", "for", "study", "in", "courses", "and", "research", "."],
                    ["terminal", "island", "is", "a", "low", "snow-covered", "island", "off", "the", "north", "tip", "of", "alexander", "island", ",", "in", "the", "bellingshausen", "sea", "west", "of", "palmer", "land", ",", "antarctic", "peninsula", "."],
                    ["among", "these", "were", "net/one", ",", "3+", ",", "banyan", "vines", "and", "novell", "'s", "ipx", "/", "spx", "."],
                    ["in", "1933\u20131970", ",", "a", "summer", "camp", "on", "south", "bass", "island", "operated", "for", "episcopal", "and", "anglican", "choristers", "."],
                    ["she", "is", "also", "the", "only", "cam", "ship", "whose", "fighter", "pilot", "died", "in", "action", "after", "his", "aircraft", "was", "launched", "from", "the", "ship", "."],
                    ["the", "department", "of", "social", "welfare", "and", "development", "(", "dswd", ")", "has", "distributed", "relief", "goods", "to", "residents", "of", "boracay", "while", "the", "island", "is", "closed", "to", "tourists", "."],
                    ["``", "rainbow", "``", "was", "scrapped", "in", "1940", "."],
                    ["it", "is", "the", "leading", "firm", "for", "the", "charlotte", "douglas", "international", "airport", "airfield", "expansion", ",", "the", "new", "dallas", "fort", "worth", "international", "airport", "southwest", "end-around", "taxiway", ",", "and", "master", "plan", "updates", "at", "philadelphia", "international", "airport", "and", "san", "antonio", "international", "airport", "."],
                    ["the", "event", "held", "at", "solberg-hunterdon", "airport", "is", "the", "largest", "summertime", "hot", "air", "balloon", "festival", "in", "north", "america", "."]
                  ],

          "label":
                  [
                    ["O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "other-biologything", "other-biologything", "O", "O", "other-biologything", "other-biologything", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "other-biologything", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O"],
                    ["O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "product-ship", "O", "product-ship", "O", "O", "O", "O", "O", "product-ship", "product-ship", "product-ship", "product-ship", "O", "O", "O"],
                    ["O", "O", "O", "O", "O", "O", "O", "O", "other-biologything", "O", "O"],
                    ["O", "O", "O", "O", "O", "O", "O", "other-biologything", "O", "other-biologything", "other-biologything", "other-biologything", "O", "O", "other-biologything", "O", "O", "other-biologything", "other-biologything", "other-biologything", "O", "O", "O", "O", "O", "O"],
                    ["O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "product-software", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O"],
                    ["product-software", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O"],
                    ["location-island", "location-island", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "location-island", "location-island", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "location-island", "location-island", "O"],
                    ["O", "O", "O", "product-software", "O", "product-software", "O", "product-software", "product-software", "O", "product-software", "product-software", "product-software", "O", "product-software", "O"],
                    ["O", "O", "O", "O", "O", "O", "O", "location-island", "location-island", "location-island", "O", "O", "O", "O", "O", "O", "O"],
                    ["O", "O", "O", "O", "O", "product-ship", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O"],
                    ["O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "location-island", "O", "O", "O", "O", "O", "O", "O", "O"],
                    ["O", "product-ship", "O", "O", "O", "O", "O", "O"],
                    ["O", "O", "O", "O", "O", "O", "O", "building-airport", "building-airport", "building-airport", "building-airport", "O", "O", "O", "O", "O", "building-airport", "building-airport", "building-airport", "building-airport", "building-airport", "O", "O", "O", "O", "O", "O", "O", "O", "O", "building-airport", "building-airport", "building-airport", "O", "building-airport", "building-airport", "building-airport", "building-airport", "O"],
                    ["O", "O", "O", "O", "building-airport", "building-airport", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O"]
                  ]
              },

  "query":
          {"word":
                  [
                    ["the", "final", "significant", "change", "in", "the", "life", "of", "the", "coco", "2", "(", "models", "26-3134b", ",", "26-3136b", ",", "and", "26-3127b", ";", "16", "kb", "standard", ",", "16", "kb", "extended", ",", "and", "64", "kb", "extended", "respectively", ")", "was", "to", "use", "the", "enhanced", "vdg", ",", "the", "mc6847t1", ",", "allowing", "lowercase", "characters", "and", "changing", "the", "text", "screen", "border", "color", "."],
                    ["the", "reno-tahoe", "international", "airport", "reno-tahoe", "international", "airport", "(", "formerly", "known", "as", "the", "reno", "cannon", "international", "airport", ")", "is", "the", "other", "major", "airport", "in", "the", "state", "."],
                    ["it", "was", "built", "by", "cole", "palen", "for", "flight", "in", "his", "weekend", "airshows", "as", "early", "as", "1967", "and", "actively", "flown", "(", "mostly", "by", "cole", "palen", ")", "within", "the", "weekend", "airshows", "at", "old", "rhinebeck", "until", "the", "late", "1980s", "."],
                    ["lambert", "land", "is", "bounded", "in", "the", "north", "by", "the", "nioghalvfjerd", "fjord", ",", "in", "the", "east", "by", "the", "greenland", "sea", "and", "in", "the", "south", "by", "the", "zachariae", "isstrom", "."],
                    ["started", "police", "operations", "with", "4", "cessna", "cu", "206g", "officially", "on", "7", "april", "1980", "with", "operations", "focused", "in", "peninsula", "of", "malaysi", "a", "."],
                    ["mysore", "airport", "is", "away", ",", "followed", "by", "kozhikode", "international", "airport", "at", "and", "bengaluru", "international", "airport", "at", "."],
                    ["the", "egg-shaped", "qaqaarissorsuaq", "island", "is", "located", "in", "tasiusaq", "bay", ",", "in", "the", "central", "part", "of", "upernavik", "archipelago", "."],
                    ["where", "they", "inserted", "nife", "hydrogenase", "into", "polypyrrole", "films", "and", "to", "provide", "proper", "contact", "to", "the", "electrode", ",", "there", "were", "redox", "mediators", "entrapped", "into", "the", "film", "."],
                    ["the", "nt-3", "protein", "is", "found", "within", "the", "thymus", ",", "spleen", ",", "intestinal", "epithelium", "but", "its", "role", "in", "the", "function", "of", "each", "organ", "is", "still", "unknown", "."],
                    ["ted", "insists", "that", "he", "will", "have", "a", "better", "chance", "at", "winning", "since", "the", "guest", "judge", ",", "tv", "presenter", "henry", "sellers", ",", "is", "staying", "at", "the", "craggy", "island", "parochial", "house", "."],
                    ["mdm2", "binds", "and", "ubiquitinates", "p53", ",", "facilitating", "it", "for", "degradation", "."],
                    ["neuraminidase", "inhibitors", "for", "human", "neuraminidase", "(", "hneu", ")", "have", "the", "potential", "to", "be", "useful", "drugs", "as", "the", "enzyme", "plays", "a", "role", "in", "several", "signaling", "pathways", "in", "cells", "and", "is", "implicated", "in", "diseases", "such", "as", "diabetes", "and", "cancer", "."], ["at", "it", "was", "long", "enough", "to", "accommodate", "the", "belle", "steamers", "that", "carried", "trippers", "along", "the", "coast", "at", "that", "time", "."],
                    ["these", "guerrilla", "sub", "missions", "originated", "at", "brisbane", "'s", ",", "capricorn", "wharf", "or", "mios", "woendi", "."],
                    ["because", "it", "was", "originally", "an", "island", "well", "within", "lake", "texcoco", ",", "iztacalco", "was", "settled", "by", "humans", "later", "than", "the", "rest", "of", "the", "valley", "of", "mexico", "."],
                    ["the", "nordic", "countries", "had", "developed", "the", "skerry", "cruiser", "classes", "and", "the", "international", "rule", "classes", "had", "adopted", "in", "1919", "a", "new", "edition", "of", "the", "rule", "which", "was", "not", "yet", "implemented", "in", "the", "countries", "."]
                  ],

          "label": [["O", "O", "O", "O", "O", "O", "O", "O", "O", "product-software", "product-software", "O", "O", "product-software", "O", "product-software", "O", "O", "product-software", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "product-software", "O", "O", "product-software", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O"], ["building-airport", "building-airport", "building-airport", "building-airport", "building-airport", "building-airport", "building-airport", "O", "O", "O", "O", "building-airport", "building-airport", "building-airport", "building-airport", "building-airport", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O"], ["O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "building-airport", "building-airport", "O", "O", "O", "O", "O"], ["location-island", "location-island", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O"], ["O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "location-island", "location-island", "location-island", "O", "O"], ["building-airport", "building-airport", "O", "O", "O", "O", "O", "building-airport", "building-airport", "building-airport", "O", "O", "building-airport", "building-airport", "building-airport", "O", "O"], ["O", "O", "location-island", "location-island", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O"], ["O", "O", "O", "other-biologything", "other-biologything", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O"], ["O", "other-biologything", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O"], ["O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "location-island", "location-island", "O", "O", "O"], ["other-biologything", "O", "O", "O", "other-biologything", "O", "O", "O", "O", "O", "O"], ["other-biologything", "other-biologything", "other-biologything", "other-biologything", "other-biologything", "O", "other-biologything", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O"], ["O", "O", "O", "O", "O", "O", "O", "O", "product-ship", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O"], ["O", "O", "O", "O", "O", "O", "product-ship", "product-ship", "O", "product-ship", "product-ship", "O", "product-ship", "product-ship", "O"], ["O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "location-island", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O"], ["O", "O", "O", "O", "O", "O", "product-ship", "product-ship", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O"]]},

  "types": ["other-biologything", "building-airport", "location-island", "product-ship", "product-software"]}

Hi! It seems like function
https://github.com/thunlp/Few-NERD/blob/3935544c428a5ed0e78adaf29a9332cc91e8bda9/util/fewshotsampler.py#L62 has a bug, which affects sampling process.

You can see "for loop" and 4 "if" inside it. If flag isvalid is set to True or False in some iterations, it can be overwritten in next iterations of the loop. So, for example, first class breaks the rule shots_count <= 2*K, but second class is new and flag isvalid will be returned as True from __valid_sample__, but sample breaks validation rules.

Thus, with this bug, we can sample samples, which are breaking sampling rules, e.g. with shots_count > 2*K.

What do you think? Does this bug affects results somehow? Maybe in the good way even.

When I try:

python train_demo.py --mode inter --lr 1e-4 --batch_size 8 --trainN 5 --N 5 --K 1 --Q 1 --train_iter 10000 --val_iter 500 --test_iter 5000 --val_step 1000 --max_length 64 --model structshot --tau 0.32

the program gets stuck on line 399 of framework.py where it tries to get next self.train_data_loader. After a minute, it raises OOM problem.

I search for a whole night and find the len function of FewShotNERDatasetWithRandomSampling returns 100000000000, which causes OOM error.

When I change it to return len(self.samples), this problem disappears. Please fix it.

Hi, thank you for sharing the data and code!

I just found that it seems that an input word is not correctly tokenized by the word tokenizer:

in the word_tokenizer.py file Each word is directly converted to token id

for raw_tokens in raw_tokens_list:
     indexed_tokens = self.tokenizer.convert_tokens_to_ids(tokens)

However, a word could be tokenized into word pieces by

for raw_tokens in raw_tokens_list:
     for word in raw_tokens:
            word_tokens = self.tokenizer.tokenize(word)

Directly converting word to token id will lead to lots of [UNK] and make the performance drop a lot.

In the following ccode https://github.com/thunlp/Few-NERD/blob/d95dba90a4c736a5310b7275d839ef7b786a46b7/util/framework.py#L537 average the F1 between batches. But this will make the evaluation results depend on the batch size chosen to conduct evaluation, right?

Hi, I attempted to use custom datasets by replacing respective data (under data/intra/train | test | dev.txt). However, I am encountering the error where I get either:

ZeroDivisionError when using proto model, or

Cannot perform max on tensor with no elements

Is there any other areas I should be amending in the data/code in order to use a custom dataset for it?

Thanks!

If I set the code to run with cpu instead of gpu, I get the above error.

This happens because in train function of FewShotNERFramework in framework.py, label is only defined if torch.cuda.isavailable is true. So then there is an assert statement a few lines later that is outside of that if block which uses label, that is assert logits.shape[0] == label.shape[0], print(logits.shape, label.shape). But further lines also depend upon label being present.

I get a RuntimeError: CUDA error: out of memory should I try to run with the GPU.

The else case for cuda not available should be dealt with better other than this assert statement in my opinon even if running this code with cpu would not be supported.

Hi, thanks for your nice work.

However, I just find the few-shot results of different versions of arXiv Papers are different. There are three groups of results: {v1,v2} , {v3,v4,v5}, {v6}.

I notice some issues mentioned that : (1) there are some imperfect implementations (such as hyperparameter selection and tokenization bug). (2) The sampled size was larger than 2K due to the data sampling bug.

May I ask if (1) is the cause of the result from {v1,v2} to {v3,v4,v5}? And is (2) (different sampled few-shot datasets) the cause of the results from {v2, V4, V5} to {v6}?
Or there are some other reasons.

This is really nice work, and thanks again for your open source and hard work : ). Look forward to your reply.

Thanks.

How can we leverage the script to create episode-data for training and test from the general custom NER data.

Though the module has the code but its a bit complex to go through it make it as utility to do this.

It would be useful to have a simple utility for 2 things:

1. Generate episode-train/test data from custom data in required format.
2. Inference script

My data is in this format:

[
{'text': 
'TN: ***************\nYour item was delivered at the front door or porch at 09:18\nam on Jan 15, 1991 in **********.\n
ABCD Tracking ADT® Available\nStatus\n✔ Delivered, Front Door \nJan 25, 1991 at 10:24 am\******************\nGet Updates', 

'spans': [{'start': 17, 'end': 39, 'label': 'TN', 'ngram': '*******************'}, 
        {'start': 142, 'end': 161, 'label': 'Carrier', 'ngram': 'ABCD Tracking ADT®'}, 
        {'start': -1, 'end': 5, 'label': 'seller', 'ngram': 'nannan'}, 
        {'start': -1, 'end': 5, 'label': 'cust', 'ngram': 'nannan'}, 
        {'start': 210, 'end': 233, 'label': 'DOS', 'ngram': '09:18\nam on Jan 15, 1991'}
        ]}
]

Please let me know if anyone has written Inference script and code to generate episode-train/test data?

@cyl628