During the training, I have around 9000+ batches of each having 32 audio segment data. After running the training for 1 epoch the mean loss and batch loss haven't changed from inf. Can you give some idea where the problem might be?

Hello,

I have trained EfficientConformerTransducerSmall model using my own data. I have used the sentencepiece tokenizer and it is also trained on my data. While decoding using best model, gready search decoding is giving around 35% WER but beam search decoding is giving higher WER( aound 40%) with beam size 5 and 16. Also with beam size 1 the WER is close to 35% which is as expected. I did the experiment with and without LM but the behaviour is same(i.e. WER is increasing with beam size higher than 1). Can you please explain why WER is inceasing for Beam Search decoding with beam size higher than 1?

When I train the ConformerCTC with the same code. The loss becomes NaN after some training epochs. After looking into it, I found that the loss is becoming NaN for a few batches but that causes accumulated loss to become NaN too. So far, I have tried the following:

1. Applied gradient clipping (both normalized and clipping by value)
2. Decreased the learning rate.
3. Applied weight clipping, to put a check if any weights are becoming too large.

None of it has worked, can you please advise on this? Thanks!

Hello,

thanks for sharing your ngram model, it does work fine with the Libre corpus in my test. Now that I want to take it one step further to train your model on my own corpus, I'd like to train my own ngram as well. I'm using the NeMo script but it doesn't seem to work. Here is the command I'm using.

python train_kenlm.py --nemo_model_file stt_en_conformer_ctc_medium.nemo --train_file MyCorpus.txt --kenlm_bin_path ../../../../kenlm/build/bin --kenlm_model_file ./isd.arpa --ngram_length 6

Here stt_en_conformer_ctc_medium.nemo is pretrained and downloaded with NeMo, and MyCorpus.txt is the collection of ground truth text. I didn't encode it as I suppose the script will do it.

What am I missing?

thanks! David

Thanks for the amazing work. I want to test the original CTC-Conformer with this code. If you have placed the pre-trained models on GD, can you please share the link like you have shared the links to Efficient Conformer model on Colab page. Thanks

Hello,

I'm following the link below to train my 6-gram LM for decoding. I use the downloaded the LibriSpeech corpus and use NeMo's CTC conformer medium model to train it. However I'm not seeing any improvement in WER compared to greedy search. The results actually became worse.

https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/asr/asr_language_modeling.html

If you could share your detailed steps for training the 6-gram LM, or your pretrained model, it would be most helpful!

David

failing while running the original notebook in

import json
import glob
import torch
import torchaudio
import IPython.display as ipd
from functions import create_model
import matplotlib.pyplot as plt

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-10-1e4ca53c14dc> in <module>()
      4 import torchaudio
      5 import IPython.display as ipd
----> 6 from functions import create_model
      7 import matplotlib.pyplot as plt

4 frames
/usr/local/lib/python3.7/dist-packages/warp_rnnt/__init__.py in <module>()
      1 import torch
----> 2 import warp_rnnt._C as core
      3 from typing import Optional, AnyStr
      4 from pkg_resources import get_distribution
      5 

ImportError: /usr/local/lib/python3.7/dist-packages/warp_rnnt/_C.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZNK2at6Tensor8data_ptrIfEEPT_v

---------------------------------------------------------------------------
NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the
"Open Examples" button below.
---------------------------------------------------------------------------

I have trained efficient conformer transducer and during inference on a CPU in a flask based web app, I see there is a memory leak at

https://github.com/burchim/EfficientConformer/blob/2f59ed2564c45c5978742fe1984cd48363064c88/models/encoders.py#L128

x, attention, hidden = block(x, mask)

The memory used at the above line while running an inference is never released, thus causing OOM is some time. Increase in memory in each iteration gets lowered but not totally removed if I use jemalloc.

When I tried running your inference script in non gready mode (beam size greater than 1), it was giving me error for not having my LM model present. Is it a compulsory requirement?

Hi Maxime @burchim , thanks for the released codes. I recently look into your implementation of RNN-Transducer and Conformer and have these questions:

1. You apply the variational noise after training 20k steps. Is there any reference about this manner? Or is it a tuned setting according to your experiments?
2. Is the broadcast of variational noise across all processes needed for distributed data parallel training?

Thanks! Huahuan.

I wanted to confirm whether the LM model is expected to be at word level or token level? Usually KenLM model is trained at word level and in our case, we are using tokenizer (n=1000), and should I need to train it at token level or word level?