Hi there! In some cases, there is an error raised during iterating over DataLoder's batches and I believe, it is happened because of offset's list length! The error is like this:

RuntimeError: Caught RuntimeError in DataLoader worker process 0. Original Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py", line 198, in _worker_loop data = fetcher.fetch(index) File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch return self.collate_fn(data) File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/collate.py", line 73, in default_collate return {key: default_collate([d[key] for d in batch]) for key in elem} File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/collate.py", line 73, in <dictcomp> return {key: default_collate([d[key] for d in batch]) for key in elem} File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/collate.py", line 55, in default_collate return torch.stack(batch, 0, out=out) RuntimeError: stack expects each tensor to be equal size, but got [150] at entry 0 and [151] at entry 1

A quick and Unprincipled solution to fix it can be adding an extra line of code to truncate the list in class NERDADataSetReader(), this is worked for me! :)

offsets = offsets[:self.max_len]

I tested the same dataset along with the same model and hyper parameters but in different versions of torch and transformers. It raises error with torch 1.81 as the following: TypeError: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]

However, it works fine in the environment in which I just had NERDA pip installed...

Hej EB,

I will be citing this work in an upcoming paper on DaCy. I have added the citation I intend to use, but feel free to change it if you feel like it doesn't fit. I will change mine accordingly. I extracted the authors from the commit history which I realize isn't always representative.

Hi, I was finally able to work on the code over the weekend, and I found the cause of the error, which was due to a tokenizer problem. In many languages, including my language (Persian), there are words and characters (abbreviations) that the tokenize() method in the tokenizers class is not able to identify, so in the face of such inputs, an empty list of word pieces is returned. In the next step, the offsets array will be expanded (by [1]) even if no word piece was identified, which eventually leads to errors in the training and evaluation process. For example, the word ۖ indicates sanctity for religious figures, which is seen in many writings but can not be identified.

Hi there - thanks for releasing your code and taking the time to read my issue!

I've just started to explore how NERDA works by trying to run the code from https://ebanalyse.github.io/NERDA/workflow/ but ran into the error of:

KeyError: 'electra'

I then changed the transformer to 'bert-base-multilingual-uncased' but now I am getting the error of:

ModuleAttributeError: 'BertModel' object has no attribute 'name_or_path'

I can't see anyone else with the same issue online and my knowledge of transformers is poor. I'd really appreciate any help you could provide.

Cheers

How could I save the trained model locally and load it next time? didn't see this in the tutorial (Perhaps it is too easy... but I couldn't find a way) Thanks in advance.

Thank you for providing such a helpful library. I have one question

I've installed from PyPI and used the Performance function. Though compute_f1_scores(y_pred, y_true, labels, **kwargs) returns a list of F1 scores, are there any ways to check recall and precision for each tag?

On the official page of Performance, I could not get information to solve the issue.

I appreciate your support.

First of all thanks a lot for making this project. You've made it super simple to train a custom model!

I experienced some issues with using a trained and saved model (on GPU) on another computer running on CPU, and I thought I'd share how to deal with it.

I could both torch.save() and torch.load() a model on my GPU pc as you write in #14. Now running my model on another pc with only CPU should be handled by providing map_location = torch.device('cpu') as pytorch write in their documentation.

So I tried that ofc using the code:

model_path = 'some_path_to_model.pt'
device = torch.device('cpu')
model = torch.load(model_path, map_location=device)

and printing model.device would return cpu. Everything seemed to be working properly.

Next when I wanted to predict_text I received this assertion error: AssertionError: Torch not compiled with CUDA enabled. Super weird since I checked that the model was on cpu.

It turned out that the NERDANetwork which is a torch.nn.Module still was casted to the 'old' GPU device. So when I was printing model.network.device cuda was returned, despite model.device returning cpu.

So my solution was to cast both the loaded model and the NERDANetwork to cpu:

model_path = 'some_path_to_model.pt'
device = torch.device('cpu')
model = torch.load(model_path, map_location=device)
model.network.device = device

I hope you'll find it useful!

I want to ask a simple question. The parameters of the model have been set before model training. What is the purpose of the validation set in model training? Thank you！

Hi there!

I have some troubles when I specify the device in NERDA model:

At the same time, when I do not specify the device parameter, everything works just fine. Looks like the trouble is in models.py here : if device != None, self.device is not initialized.

Is the evaluation in evaluate_performance function (models.py) done at Token level or entity level? It seemed like it is done on the token level, from the image in the workflow documentation, but the comments in the code mention entity level.

We are using NERDA for a couple things, but we currently get this error using NERDA as a dependency.

× python setup.py egg_info did not run successfully.
    │ exit code: 1
    ╰─> [18 lines of output]
        The 'sklearn' PyPI package is deprecated, use 'scikit-learn'
        rather than 'sklearn' for pip commands.
        
        Here is how to fix this error in the main use cases:
        - use 'pip install scikit-learn' rather than 'pip install sklearn'
        - replace 'sklearn' by 'scikit-learn' in your pip requirements files
          (requirements.txt, setup.py, setup.cfg, Pipfile, etc ...)
        - if the 'sklearn' package is used by one of your dependencies,
          it would be great if you take some time to track which package uses
          'sklearn' instead of 'scikit-learn' and report it to their issue tracker
        - as a last resort, set the environment variable
          SKLEARN_ALLOW_DEPRECATED_SKLEARN_PACKAGE_INSTALL=True to avoid this error

The issue stems from adding the deprecated sklearn instead of scikit-learn in your setup file at line 21:

https://github.com/ebanalyse/NERDA/blob/ae45d7e5368059721d1073384201433ea7a6e820/setup.py#L21

Would you please change the dependency to scikit-learn instead

Changes:

1. Change load_network_from_file, and save_network methods. It allows us to save and load tokenizer now. Save network before: model.save_network(model_path="model.bin") Save network now: model.save_network(output_dir="models/") -- Load network before: model.load_network_from_file(model_path="model.bin") Load network now: model.load_network_from_file(model_path="models/model.bin", tokenizer_path="models/tokenizer/")

2. Typo change: AVG_MICRO --> AVG_MACRO (from previous PL: https://github.com/ebanalyse/NERDA/pull/33)

3. Added version number to package (from previous PL: https://github.com/ebanalyse/NERDA/pull/34)

4. Adapting preprocessing for some models of HF (from previous PL: https://github.com/ebanalyse/NERDA/pull/36, but removed line 106)

Change this:

msg = f'Sentence #{item} length {len(tokens)} exceeds max_len {self.max_len} and has been truncated'

to

msg = f'Sentence #{item} length {len(tokens)} exceeds max_len {self.max_len} - 2 and has been truncated, note that two tokens are used to surround the sentence with the [CLS] and [SEP] token'

Since the warning Sentence 4 length 511 exceeds max_len 512 and has been truncated doesn't make sense.

ValueError: Found input variables with inconsistent numbers of samples: [18734, 18733]

I am able to get F1 scores when i pass test_dict to evaluate function as model.evaluate_performance(test_dict) but if i pass "True"as a parameter to model.evaluate_performance(test_dict,True) i am getting a ValueError. I crossed checked the test_dict manually there are total 18734 samples in y_test so why is the function missing out one value in y_pred that is the predicted number of samples ?.

If someone passes the outside_tag as part of the tag scheme they would probably get something like:

[Expected input batch_size (324) to match target batch_size (4)]

That is because the scheme_tag would inflate the batch size by some extra bytes (the direct portion of the outside tag from the number of other schema tags). This can easily be fixed by putting in the training.py [138:0]:

    tag_complete = list(set([tag_outside] + tag_scheme))