Seungwon, hello.

My name is Vladimir. I am a researcher at Speech Technology Center, Russia, St. Petersburg. Your implementation of the VoiceFilter algorithm (https://github.com/mindslab-ai/voicefilter) is very interesting to me and my colleagues. Unfortunately, we could not get the SDR metric dynamics like yours, using your code with the standard settings in the default.yaml file. SDR converged to 4.5 dB after 200k iterations (see figure below), but not to 10 dB after 65k as in your results. Could you tell us your training settings, as well as the neural network architecture that you used to get your result?

Our python environment:

1. tqdm (ver. 4.32.1);
2. numpy (ver. 1.16.3);
3. torch (ver. 1.1.0);
4. pyyaml (ver. 5.1);
5. librosa (ver. 0.6.3);
6. mir_eval (ver. 0.5);
7. matplotlib (ver. 3.1.0);
8. tensorboardX (ver. 1.7);
9. ffmpeg (ver. 4.1.3);
10. ffmpeg_normalize (1.14.0);
11. python (ver. 3.6).

We use four Nvidia GeForce GTX 1080 Ti when training one VoiceFilter's model. Subsets train-clean-100, train-clean-360 and train-other-500 from LibriSpeech dataset are used to train VoiceFilter's model and dev-clean is used to test VoiceFilter's model. We use the pretrained d-vector model to encode the target speaker.

We used your default configuration file:

audio:
  n_fft: 1200
  num_freq: 601
  sample_rate: 16000
  hop_length: 160
  win_length: 400
  min_level_db: -100.0
  ref_level_db: 20.0
  preemphasis: 0.97
  power: 0.30

model:
  lstm_dim: 400
  fc1_dim: 600
  fc2_dim: 601

data:
  train_dir: 'path/to/train/data'
  test_dir: 'path/to/test/data'
  audio_len: 3.0

form:
  input: '*-norm.wav'
  dvec: '*-dvec.txt' 
  target:
    wav: '*-target.wav'
    mag: '*-target.pt'
  mixed:
    wav: '*-mixed.wav'
    mag: '*-mixed.pt'

train:
  batch_size: 8
  num_workers: 16
  optimizer: 'adam'
  adam: 0.001
  adabound:
    initial: 0.001
    final: 0.05
  summary_interval: 1
  checkpoint_interval: 1000

log:
  chkpt_dir: 'chkpt'
  log_dir: 'logs'

embedder:
  num_mels: 40
  n_fft: 512
  emb_dim: 256
  lstm_hidden: 768
  lstm_layers: 3
  window: 80
  stride: 40

The neural network architecture was standard and followed your implementation.

I tried to try the trained model on a single input and it gave OOM on GCP with 1 Nvidia P100. RuntimeError: CUDA out of memory. Tried to allocate 4.66 GiB (GPU 0; 15.90 GiB total capacity; 14.37 GiB already allocated; 889.81 MiB free; 19.21 MiB cached) The file size of the mixed wav(19 MB) file was about 5 minutes and for reference file was 11 seconds. I don't know why it shows 14.37 GiB allocated when not even training. I tried to restart the instance but it did not help. Can you please suggest a way to reduce the memory required while Inference? Thank you!

Hi, if in case I wanted to use another dataset of audio files for the training and the test (not the one used here) the embedder.pt that I have to insert when I run "trainer.py" as I can generate it or which one I have to use ? Thank you

Thank you for your great job! I have a question when I tried to run the project. I set 'N' as my CPU core number , then I input 'chmod a+x normalize-resample.sh'. However, after I input './normalize-resample.sh ', there is no output on the command line. Is this normal? Furthermore, what is the function of this script?

Next, copy utils/normalize-resample.sh to root directory of unzipped data folder. Then:

vim normalize-resample.sh # set "N" as your CPU core number.
chmod a+x normalize-resample.sh
./normalize-resample.sh # this may take long

Looking for your reply!

Dear @seungwonpark

First of all, I would like to thank you for great open source. I would like to test your nice code and I tried to train voice filter.

But i get the problem with SDR. When i saw SDR graph in voicefilter github, SDR value from 2 to 10dB. But in my case, SDR value is from -0.8 to 1.2.

I am trying to find the cause of the problem but I can not find it.

Can you help me to find the cause of problem?

I used the default yaml and generator.py. ( train-clean-100, train-clean-360, dev-clean are used to train)

Could you let me know what i can check?

Thanks you!

Hi, I'd like to use this voice filtering in real-time. Would it be possible to modify the inference code to run the model in real time for audio PCM data?

Newer packages of those libraries have changed things and the methods used in the code are no longer available. Setting specific versions in the requirements file avoids installing those versions.

Yes, it gives import error on my new venv.

p.s. Doesn't know if I'm running it wrong, it took 3 hours to run only 1.4k updates on my 1080ti. And the SDR is even negative number.

Hello, I'm a master student in ITMO university in Saint-Petersburg, Russia.

Could you explain me please, what exactly this model implemenation do? As for me (variant 1) it takes as input mixed sound of voice of a person A and voice of a person B and clear voice A, the same as in mixed one and trying to extract it from the mixed one. (that is really strange because it is useless) And in the paper (variant 2) it is said that it should take the mixed one and clear voice of the target person but NOT the same sound as in mixed one! And this is the point.

When I tried to look at train test, made by generator, I found out that in every example of ******-mixed.wav there is ******-target.wav with another voice! (but not another phrase of target person as I thought it should be)

Am I right? Or what's going on here?

Waiting for your answer, thank you!

It's 1.0.2 on pypi and 1.1.0 on the way.

p.s. It would be great if you plan to migrate from requirements.txt to pipenv for versioning. It would help you resolve version and lock them for you automatically.

Hi, it's me again:) Because of insufficient computer storage,I skipped the following step:

Preprocess wav files

In order to boost training speed, perform STFT for each files before training by:

python generator.py -c [config yaml] -d [data directory] -o [output directory] -p [processes to run]

This will create 100,000(train) + 1000(test) data. (About 160G)

Then I downloaded embedder.pt, train-clear-100.tar.gz and dev-clean.tar.gz. I unziped tar.gz files and put those unzipped file folders to the root directory of voicefilter.I also specifying train_dir, test_dir at config.yaml, such as:

  train_dir: '/home/../voicefilter/train/train-clean-100'
  test_dir: '/home/../voicefilter/dev/dev-clean'

After that, when I enter this instruction:

python trainer.py -c [config yaml] -e [path of embedder pt file] -m [name]

Error prompts pop up on the screen:AssertionError: no training file found

I want to know which step I made a mistake, or what configuration was missing? Thanks! REALLY looking forward to your reply!

Hello, I have two questions about the implementation.

1. I cannot reproduce the results reported in the README. I have trained for around > 400k steps on Librispeech 360h + 100h clean dataset, using the embedder provided in this repo. However, I can only obtain up to a maximum SDR of 5.5.

To obtain data from the Librispeech 360h + 100h, I generate the mixed audios for 360h and 100h separately, then add them together in another folder. Is this the right way when I want to use more data to train the voice filter module?

2. I got worse results when retraining the speaker embedding I retrained the embedder using the following repo: Speaker verification on 3 datasets: Librispeech, VoxCeleb1, VoxCeleb2.

Theoretically, I expect the voice filter module will benefit from the embedder trained on more data, but the results got even worse. Can you share how you train this embedder?

Thank you in advance!

I have noticed that there is amp_to_db function in wav2spec, which means that the input of model is db-scaled magnitude. I s this right? Since this is not mentioned in related paper.

hi~ i meet with a problem when doing ./normalize-resample.sh , it seems that wav in /tmp did not exit, i try to fix it but failed, does anyone know where the problem is ? i also run the commend " ffmpeg-normalize 1.wav -o 1-norm.wav" to test this normalize tool , and occured the same question ; how to make this normalize tool (ffmpeg-normalize） work？ yingyingying

Hi,

Thank you for publishing your code! I am encountering a training problem. As an initial phase I have tried to train only on 1000 samples from LibriSpeech train-clean-100 dataset. I am using the default configuration as was published in your VoiceFilter repo. The only difference is that I used batch size of 6 due to memory limitations. Is it possible that the problem is related to the small batch size that I use?

Another question is related to the generation of the training and testing sets. I have noticed that there is an option to use a VAD for generating the training set but by default it is not used. What is the best practice? to use the VAD or not?

I appreciate your help!

Hi, seungwonpark,

I was trying to use Google's posted test list for LibriSpeech to reproduce their results. But I can not even get their initial mean SDR (10.1 dB in their paper). I got only 1.5 dB. I am wondering have you tried their list and got around 10.1 dB for mean SDR before applying voice filter?

Thank you so much.