Whenever i run the model (facebook/bart-large-cnn), or even the bart-base model to try to summarize text, using the code below, i keep running into this error, and i'm honestly stuck, so any help would be greatly appreciated.

Code:

from fastBart import export_and_get_onnx_model, OnnxBart, quantize
from transformers import AutoTokenizer
from pathlib import  Path
from fastBart.ort_settings import get_onnx_runtime_sessions
import os
from fastBart.onnx_exporter import (
    generate_onnx_representation)

model_name = 'facebook/bart-large-cnn'

# generate onnx model
model_paths = generate_onnx_representation(model_name)
model_sessions = get_onnx_runtime_sessions(tuple(model_paths))
model_paths = tuple(model_paths)

# after model gotten
# name = 'bart-large-cnn'
# custom_output_path = './models-bart/'
# model_paths = [os.path.join(custom_output_path,name+"-"+x) for x in ['encoder.onnx','decoder.onnx','init-decoder.onnx']]
# model_paths = tuple(model_paths)


model_sessions = get_onnx_runtime_sessions(model_paths)
model = OnnxBart(model_name, model_sessions)



tokenizer = AutoTokenizer.from_pretrained(model_name)

input = """Since being shared on Reddit, the 21-second video has garnered more than 35,000 upvotes. People also filled the comment section with adorable comments.
A Reddit user said that he liked the little cat staring at the mama dog for a moment. Another liked the calmness and patience of the mama dog.
“This is soooo cute,” commented a third Reddit user. Many users also added that humans should learn kindness from these animals.
Well, this is not the first time that a kitten was feeding on a nursing dog.Last year, a similar heartwarming video had surfaced 
from a remote village in Nigeria that showed a little feline feeding on milk from a mama dog. The clip, which had gone viral, has 
so far accumulated over 1 million views. Twitter users showered love for the dog and the little kitten. The 32-second clip was shared 
by Reuters on its official Twitter handle with the caption, “It's a most unusual sight: a kitten was spotted feeding on milk from a nursing dog in a remote village in Nigeria.”
"""

token = tokenizer(input, return_tensors='pt',  truncation=True)
tokens = model.generate(input_ids=token['input_ids'],attention_mask=token['attention_mask'],num_beams=4)

output = tokenizer.decode(tokens.squeeze(), skip_special_tokens=True)
print(output)

Error:

Traceback (most recent call last):
File "/Users/zeke/Documents/Github/SumanyWeb/sumany/summarize/fast-Bart/bartoptimize.py", line 34, in <module>
    tokens = model.generate(input_ids=token['input_ids'],attention_mask=token['attention_mask'],num_beams=4)
File "/usr/local/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/transformers/generation_utils.py", line 1162, in generate
    inputs_tensor, model_input_name, model_kwargs = self._prepare_model_inputs(inputs, bos_token_id, model_kwargs)
File "/usr/local/lib/python3.9/site-packages/transformers/generation_utils.py", line 412, in _prepare_model_inputs
    and self.encoder.main_input_name != self.main_input_name
File "/usr/local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1207, in __getattr__
    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'BartEncoder' object has no attribute 'main_input_name'

Hi @siddharth-sharma7

Thank you for providing fast-bart. It has made my life much easier.

I find the bart-onnx-quantized model 2-3x faster than the Pytorch model. However, when the sequence length is long (~500 tokens), the onnx-based model is 1.5-2x slower.

I also find a similar problem for T5-onnx model that has been discussed at https://github.com/microsoft/onnxruntime/issues/6835#:~:text=the%20converted%20t5%20onnx%20model,and%20higher%20beam%2Dsearch%20number.

Just wondering if we're facing the same issue here.

Hi @siddharth-sharma7 , You package is great and very easy to use, but I'm unable to figure out how to actually use CUDAExecutionProvider, and use gpu for inferencing. Whenever I provide the providers=['CUDAExecutionProvider'], the model is still not being loaded to gpu and inferencing still happens in cpu.

Thank you very much for your work, very helpful! Now I can convert BART model to ONNX model, and the output of the two is consistent, but I would like to ask whether you have tried to deploy onNX model to Tensorrt, so far I have been able to run on Tensorrt, but the result of tensorRT deployment is not consistent with that of ONNX model; (