First, congratulations on this great job.

I'm trying to replicate your experiments using Google Colab. I managed to do the training, but when I do the inference it triggers an error: RuntimeError: result type Float can't be cast to the desired output type Long

Could you help me with something that can resolve the above error?

Thank you for your attention! Congratulations again for the work!

From Table.9 (under section 6.2) in the paper, I am confused regarding the difference between the S2G->G2T model and the Transformer model (S2G2T?). Could you please clarify the exact difference between these two?

Just wanted to confirm that STMC-Transformer and STMC-RNN are both S2G->G2T models and NOT S2G2T models. My understanding of these terms is as follows: S2G->G2T: Where S2G model is trained first, and then, the G2T model is trained on the predictions of the S2G model. S2G2T: Where S2G & G2T models are trained in a joint manner. Please correct me if I am wrong.

Dear sir, I want do the experiment on colab follow your readme.md procedures, but I have encounter some weried problems. I first run the following commands: ! pip install virtualenv ! virtualenv --python=python3 venv ! source venv/bin/activate ! pip install torch==1.6.0 ! git clone https://github.com/kayoyin/transformer-slt.git ! pip install -r requirements.txt ! python setup.py install

that's fun. but after I copy your onmt-preprocess command which is

onmt_preprocess -train_src data/phoenix2014T.train.gloss -train_tgt data/phoenix2014T.train.de -valid_src data/phoenix2014T.dev.gloss -valid_tgt data/phoenix2014T.dev.de -save_data data/dgs -lower

It returns the error:

File "", line 1 onmt_preprocess -train_src data/phoenix2014T.train.gloss -train_tgt data/phoenix2014T.train.de -valid_src data/phoenix2014T.dev.gloss -valid_tgt data/phoenix2014T.dev.de -save_data data/dgs -lower ^ SyntaxError: invalid syntax

I really don't know why it happens, could you help me ?

Hi, I have read your work carefully and it seems to me well done, congratulations. I am very interested in being able to replicate your results in an end to end manner, and, subsequently, i would like to adapt the work to an Italian Sign Language dataset. Your work is based on two steps, the first is the translation from sign language to glosses through the STMC network. Subsequently, the glosses are translated into text through the transformers. Is there the possibility to share the code regarding the first part in which the STMC network is defined and trained? I'm really interested in the mechanism that takes videos in and translates them into glosses in a continuous way.

Thanks, Enrico

I am trying to run the server howewer a JSON file is required in def start(self, config_file). I was wondering whether you could provide an example of this file so I am able to see the structure and type of data that is needed. Thank you very much.

Although the size of the train vocabulary is 1066 In your paper, the size of the train vocabulary is 1232 which I built from "phoenix2014T.train.gloss".

Here's the code to find the train vocabulary size.

vocab = {}

with open("phoenix2014T.train.gloss", "r") as f:
    lines = f.readlines()

for sentence in lines:
    for word in sentence.strip().split(" "):
        if word in vocab:
            continue
        vocab[word] = len(vocab)
        
print(len(vocab))

Hi, thanks for this open source repo. I am trying to execute the commands given in the README. I am receiving the following error. Any help would be appricated.

(py36torch) E:\transformer-slt>python  train.py -data data/dgs -save_model model -keep_checkpoint 1 -layers 2 -rnn_size 512 -word_vec_size 512 -transformer_ff 2048 -heads 8  -encoder_type transformer -decoder_type transformer -position_encoding -max_generator_batches 2 -dropout 0.1 -early_stopping 3 -early_stopping_criteria accuracy ppl -batch_size 2048 -accum_count 3 -batch_type tokens -normalization tokens -optim adam -adam_beta2 0.998 -decay_method noam -warmup_steps 3000 -learning_rate 0.5 -max_grad_norm 0 -param_init 0  -param_init_glorot -label_smoothing 0.1 -valid_steps 100 -save_checkpoint_steps 100 -world_size 1 -gpu_ranks 0
Traceback (most recent call last):
  File "train.py", line 6, in <module>
    main()
  File "E:\transformer-slt\onmt\bin\train.py", line 204, in main
    train(opt)
  File "E:\transformer-slt\onmt\bin\train.py", line 35, in train
    vocab = torch.load(opt.data + '.vocab.pt')
  File "E:\Envs\py36torch\lib\site-packages\torch\serialization.py", line 584, in load
    with _open_file_like(f, 'rb') as opened_file:
  File "E:\Envs\py36torch\lib\site-packages\torch\serialization.py", line 234, in _open_file_like
    return _open_file(name_or_buffer, mode)
  File "E:\Envs\py36torch\lib\site-packages\torch\serialization.py", line 215, in __init__
    super(_open_file, self).__init__(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: 'data/dgs.vocab.pt'

When i execute the testing command i am getting the following error:

(py36torch) E:\transformer-slt>python translate.py -model model [model2 model3 ...] -src data/phoenix2014T.test.gloss -output pred.txt -gpu 0 -replace_unk -beam_size 4
Traceback (most recent call last):
  File "translate.py", line 6, in <module>
    main()
  File "E:\transformer-slt\onmt\bin\translate.py", line 48, in main
    translate(opt)
  File "E:\transformer-slt\onmt\bin\translate.py", line 18, in translate
    translator = build_translator(opt, report_score=True)
  File "E:\transformer-slt\onmt\translate\translator.py", line 28, in build_translator
    fields, model, model_opt = load_test_model(opt)
  File "E:\transformer-slt\onmt\decoders\ensemble.py", line 130, in load_test_model
    onmt.model_builder.load_test_model(opt, model_path=model_path)
  File "E:\transformer-slt\onmt\model_builder.py", line 96, in load_test_model
    map_location=lambda storage, loc: storage)
  File "E:\Envs\py36torch\lib\site-packages\torch\serialization.py", line 584, in load
    with _open_file_like(f, 'rb') as opened_file:
  File "E:\Envs\py36torch\lib\site-packages\torch\serialization.py", line 234, in _open_file_like
    return _open_file(name_or_buffer, mode)
  File "E:\Envs\py36torch\lib\site-packages\torch\serialization.py", line 215, in __init__
    super(_open_file, self).__init__(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: 'model'

Please suggest the solution

I am assuming you used vanilla embeddings in your first ablation experiment on the number of encoder-decoder layers. Moreover, the paper states that it uses #layers=2 for further experiments. In that case, shouldn't the results in Table-3 (for #layers = 2) match the results in Table-5 (for vanilla embedding)? For example, the BLEU-4 score (for test set) in Table-3 is 21.65, and in Table-5, is 22.22. Shouldn't these not be different?

I'm trying to convert the pytorch model to tflite model. But I'm facing issue in providing the dummy model input to torch.onnx.export() method. Do you know what can be the dummy model input ?

Here is how my code look like:

`import torch.nn as nn import torch.onnx import torchvision import torch from onmt.model_builder import build_base_model from onmt.utils.parse import ArgumentParser

checkpoint = torch.load('model_step_1600.pt')

model_opt = ArgumentParser.ckpt_model_opts(checkpoint['opt']) ArgumentParser.update_model_opts(model_opt) ArgumentParser.validate_model_opts(model_opt) vocab = checkpoint['vocab'] fields = vocab

model = build_base_model(model_opt, fields, None, checkpoint) model.eval()

dummy_input = torch.randn(1, 3, 224, 224, requires_grad=True) # dummy_input = torch.from_numpy(X_test[0].reshape(1, -1)).float().to(device)

torch.onnx.export(model, dummy_input, 'model_simple.onnx') `