A Low Complexity Speech Enhancement Framework for Full-Band Audio (48kHz) based on Deep Filtering.

Hendrik Schröter

Last update: Dec 25, 2022

Related tags

Deep Learning DeepFilterNet

Overview

DeepFilterNet

A Low Complexity Speech Enhancement Framework for Full-Band Audio (48kHz) based on Deep Filtering.

libDF contains Rust code used for data loading and augmentation.
DeepFilterNet contains Python code including a libDF wrapper for data loading, DeepFilterNet training, testing and visualization.
models contains DeepFilterNet model weights and config.

Usage

System requirements are cargo and pip (Rust and Python package managers). Usage of a conda or virtualenv recommended. This framework is currently only tested under Linux.

Installation of python dependencies and libDF:

cd path/to/DeepFilterNet/  # cd into repository
# Recommended: Install or activate a python env.
pip install maturin  # Used to compile libDF and load
maturin build --release -m DeepFilterNet/Cargo.toml  # Build python wheel
# Install python wheel. Make sure to specify the correct DeepFilterNet and python version
pip install target/wheels/DeepFilterNet-0.1.0-cp39-cp39-linux_x86_64.whl
# Optional: Install cuda version of pytorch from pytorch.org
pip install -r requirements.txt  # Install remaining dependencies

To enhance noisy audio files using DeepFilterNet run

# usage: enhance.py [-h] [--output-dir OUTPUT_DIR] model_base_dir noisy_audio_files [noisy_audio_files ...]
python DeepFilterNet/df/enhance.py models/DeepFilterNet/ path/to/noisy_audio.wav

License

DeepFilterNet is free and open source! All code in this repository is dual-licensed under either:

MIT License (LICENSE-MIT or http://opensource.org/licenses/MIT)
Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)

at your option. This means you can select the license you prefer!

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Comments

when i training with MS-SNSD dataset, and prepared the data correctly.it aways get error: RuntimeError: DF dataloader error: ThreadJoinError("Any { .. }")

config, and logs as below：

cat base_dir/config.ini [train] seed = 42 device = model = deepfilternet3 mask_only = False df_only = False jit = False batch_size = 1 batch_size_eval = 0 overfit = False num_workers = 0 max_sample_len_s = 5.0 num_prefetch_batches = 32 global_ds_sampling_f = 1.0 dataloader_snrs = -5,0,5,10,20,40 batch_size_scheduling = max_epochs = 10 validation_criteria = loss validation_criteria_rule = min early_stopping_patience = 5 start_eval = False

[df] sr = 48000 fft_size = 960 hop_size = 480 nb_erb = 32 nb_df = 96 norm_tau = 1 lsnr_max = 35 lsnr_min = -15 min_nb_erb_freqs = 2 df_order = 5 df_lookahead = 0 pad_mode = input

[deepfilternet] conv_lookahead = 0 conv_ch = 16 conv_depthwise = True convt_depthwise = True conv_kernel = 1,3 conv_kernel_inp = 3,3 emb_hidden_dim = 256 emb_num_layers = 2 emb_gru_skip = none df_hidden_dim = 256 df_gru_skip = none df_pathway_kernel_size_t = 1 enc_concat = False df_num_layers = 3 df_n_iter = 1 linear_groups = 1 enc_linear_groups = 16 mask_pf = False

[distortion] p_reverb = 0.2 p_bandwidth_ext = 0.0 p_clipping = 0.0 p_zeroing = 0.0 p_air_absorption = 0.0

[optim] lr = 0.0005 momentum = 0 weight_decay = 0.05 optimizer = adamw lr_min = 1e-06 lr_warmup = 0.0001 warmup_epochs = 3 lr_cycle_mul = 1.0 lr_cycle_decay = 0.5 lr_cycle_epochs = -1 weight_decay_end = -1

[maskloss] factor = 0 mask = iam gamma = 0.6 gamma_pred = 0.6 f_under = 2

[spectralloss] factor_magnitude = 0 factor_complex = 0 factor_under = 1 gamma = 1

[multiresspecloss] factor = 0 factor_complex = 0 gamma = 1 fft_sizes = 512,1024,2048

[sdrloss] factor = 0

[localsnrloss] factor = 0.0005

logs: python df/train.py dataset_small.cfg data-dir-small/ base_dir_small/ 2022-12-21 15:31:00 | INFO | DF | Running on torch 1.12.0+cu102 2022-12-21 15:31:00 | INFO | DF | Running on host research-bj-green-x4-0 fatal: not a git repository (or any of the parent directories): .git 2022-12-21 15:31:00 | INFO | DF | Loading model settings of base_dir_small 2022-12-21 15:31:00 | INFO | DF | Running on device cuda:0 2022-12-21 15:31:00 | INFO | DF | Initializing model deepfilternet3 2022-12-21 15:31:03 | INFO | DF | Initializing dataloader with data directory data-dir-small/ 2022-12-21 15:31:03 | INFO | DF | Loading HDF5 key cache from .cache_dataset_small.cfg 2022-12-21 15:31:03 | INFO | DF | Start train epoch 0 with batch size 1 2022-12-21 15:31:04 | INFO | DF | [0] [0/9] | loss: 0.28417 | lr: 1.000E-04 thread '' panicked at 'assertion failed: k <= self.len()', /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/slice/mod.rs:3094:9 note: run with RUST_BACKTRACE=1 environment variable to display a backtrace 2022-12-21 15:31:04 | ERROR | DF | An error has been caught in function '', process 'MainProcess' (29909), thread 'MainThread' (140548370358784): Traceback (most recent call last):

File "df/train.py", line 638, in main() └ <function main at 0x7fd3e3bf40d0>

File "df/train.py", line 248, in main train_loss = run_epoch( └ <function run_epoch at 0x7fd3e3bfc040>

File "df/train.py", line 355, in run_epoch for i, batch in enumerate(loader.iter_epoch(split, seed)): │ │ │ │ │ └ 0 │ │ │ │ └ 'train' │ │ │ └ <function PytorchDataLoader.iter_epoch at 0x7fd3e3bf4e50> │ │ └ <libdfdata.torch_dataloader.PytorchDataLoader object at 0x7fd3dc59a5b0> │ └ Batch of size 1: │ SNRs: 40 │ Gains: -6 │
└ 0

File "/home/xxx/code/deepfilter/DeepFilterNet-0.4.0/pyDF-data/libdfdata/torch_dataloader.py", line 283, in iter_epoch batch = self._get_batch() │ └ <function PytorchDataLoader._get_batch at 0x7fd3e3bf4d30> └ <libdfdata.torch_dataloader.PytorchDataLoader object at 0x7fd3dc59a5b0>

File "/home/xxx/code/deepfilter/DeepFilterNet-0.4.0/pyDF-data/libdfdata/torch_dataloader.py", line 232, in _get_batch _, batch = self.data_queue.get() │ │ └ <function PytorchDataLoader._get_worker_queue_dummy.._Queue.get at 0x7fd3da327310> │ └ <libdfdata.torch_dataloader.PytorchDataLoader._get_worker_queue_dummy.._Queue object at 0x7fd3dca65640> └ <libdfdata.torch_dataloader.PytorchDataLoader object at 0x7fd3dc59a5b0>

File "/home/xxx/code/deepfilter/DeepFilterNet-0.4.0/pyDF-data/libdfdata/torch_dataloader.py", line 214, in get self.loader.cleanup() │ │ └ <method 'cleanup' of 'builtins._FdDataLoader' objects> │ └ <builtins._FdDataLoader object at 0x7fd3dc730100> └ <libdfdata.torch_dataloader.PytorchDataLoader object at 0x7fd3dc59a5b0>

RuntimeError: DF dataloader error: ThreadJoinError("Any { .. }")

opened by BWMa 27

Question for training

Hi Rikorose,

Sorry to bother you again, I try to generate data and train the model according to the training part.

I generated the training_set.txt (just select 10 files for test.) for speech and made the hdf5.(and so on for noise). Use python df/prepare_data.py --sr 48000 speech training_set.txt TRAIN_SET_SPEECH.hdf5.

~/DeepFilterNet/wav/dataset/oblomov_s009036.wav
~/DeepFilterNet/wav/dataset/oblomov_s009040.wav  
~/DeepFilterNet/wav/dataset/oblomov_s009033.wav     
~/DeepFilterNet/wav/dataset/oblomov_s009037.wav    
~/DeepFilterNet/wav/dataset/oblomov_s009041.wav  
~/DeepFilterNet/wav/dataset/oblomov_s009034.wav    
~/DeepFilterNet/wav/dataset/oblomov_s009038.wav     
~/DeepFilterNet/wav/dataset/oblomov_s009042.wav  
~/DeepFilterNet/wav/dataset/oblomov_s009035.wav     
~/DeepFilterNet/wav/dataset/oblomov_s009039.wav

Generate the dataset.cfg as shown below,

{
 "train": [
    [
      "~/DeepFilterNet/DeepFilterNet-github/DeepFilterNet/hdf5/TRAIN_SET_SPEECH.hdf5",
      1.0
    ],
    [
      "~/DeepFilterNet/DeepFilterNet-github/DeepFilterNet/hdf5/TRAIN_SET_NOISE.hdf5",
      1.0
    ]
  ],
  "valid": [
    [
      "~/DeepFilterNet/DeepFilterNet-github/DeepFilterNet/hdf5/TRAIN_SET_SPEECH.hdf5",
      0.2
    ],
    [
      "~/DeepFilterNet/DeepFilterNet-github/DeepFilterNet/hdf5/TRAIN_SET_NOISE.hdf5",
      0.2
    ]
  ],
  "test": [
    [
      "~/DeepFilterNet/DeepFilterNet-github/DeepFilterNet/hdf5/TRAIN_SET_SPEECH.hdf5",
      0.2
    ],
    [
      "~/DeepFilterNet/DeepFilterNet-github/DeepFilterNet/hdf5/TRAIN_SET_NOISE.hdf5",
      0.2
    ]
  ]
}

Encounter some error as shown in the figure below,

In addition, I have some questions:

In the command python df/train.py dataset.cfg ~/wav_folder/ ./base_dir/, -Is data_dir a wav folder or an hdf5 folder? (I think is hdf5 folder) -Can base_dir/ not exist? (But we need to give config.ini, so here I enter pertrained_model/ and delete .ckpt)
I found that the log says dataloader len:0, is this a problem?
I remove the all 'df.' of each file while import (ex. from df.config import ... -> from config import ...), otherwise it will cause an import error.

Thanks,

opened by aaronhsueh0506 25

Reproducing results

Hi,

I had tried to re-train the deepfilternet model using the DNS-3 challenge dataset mentioned in your work.

I don't have the additional 10k IR. However, the other dataset remains the same.

On VCTK test set, using the config.ini in the pre-trained model as my training config, my "best model" on validation gives PESQ score of 2.60. It is much lower than 2.81 from the pre-trained model.

In config.ini, Adamw is used, while in the paper Adam as optimizer is mentioned.

Do you think any other factors would result in such a performance drop?

Could you clarify on the 3 s sample for training? Suppose the DNS-3 sample has 10 s in a sample, do I need to split it into 3 s segments so as to utilize the entire train clip? Or just use the first 3 seconds of the clip? Alternatively, is random 3 s generated on-the-fly while training?

In the hdf5 setup, does the speech/noise/rir need to have sample number of samples? Or is the noise and RIR sampled randomly from a list? For example, if the speech list has 1000 samples, noise list is 100 samples and rir list is 100 samples, is it okay? or should it be 1000 speech, 1000 noise, 1000 rir? Is it needed to make the duration of speech and noise samples to be the same?

How about the reverberation parameter p_reverb = 0.05? The data augmentation is performed by default or any other config is needed? conv_lookahead = 2 in config.ini. But the paper mentions "look-ahead of l = 1 frame for both DF as well as in the DNN convolutions".
stale

opened by rohithmars 15

RuntimeError: DF dataloader error: ThreadJoinError("Any { .. }") when attempting to train DeepFilterNet

I am trying to train DeepFilterNet but I am running into the following error when training: RuntimeError: DF dataloader error: ThreadJoinError("Any { .. }").

What I did

I installed DeepFilterNet via PyPI using:

pip3 install torch torchvision torchaudio
pip install deepfilternet[train]

Then I generated the following HDF5 dataset files:

TEST_SET_NOISE.hdf5
TEST_SET_SPEECH.hdf5
TRAIN_SET_NOISE.hdf5
TRAIN_SET_SPEECH.hdf5
VALID_SET_NOISE.hdf5
VALID_SET_SPEECH.hdf5

and placed them into the same directory .../data_folder.

I also created a config.ini file with the following contents:

config.ini

[deepfilternet]
emb_hidden_dim = 256
df_hidden_dim = 256
df_num_layers = 3
conv_ch = 16
conv_lookahead = 0
conv_depthwise = True
convt_depthwise = True
conv_kernel = 1,3
conv_kernel_inp = 3,3
emb_num_layers = 2
emb_gru_skip = none
df_gru_skip = none
df_pathway_kernel_size_t = 1
enc_concat = False
df_n_iter = 1
linear_groups = 1
enc_linear_groups = 16
mask_pf = False
gru_type = grouped
gru_groups = 1
group_shuffle = True
dfop_method = real_unfold
df_output_layer = linear

[df]
fft_size = 960
nb_erb = 32
nb_df = 96
sr = 48000
hop_size = 480
norm_tau = 1
lsnr_max = 35
lsnr_min = -15
min_nb_erb_freqs = 2
df_order = 5
df_lookahead = 0
pad_mode = input

[train]
model = deepfilternet2
batch_size = 16
batch_size_eval = 16
num_workers = 4
overfit = false
lr = 0.001
max_epochs = 30
seed = 42
device = cuda:1
mask_only = False
df_only = False
jit = False
max_sample_len_s = 5.0
num_prefetch_batches = 32
global_ds_sampling_f = 1.0
dataloader_snrs = -5,0,5,10,20,40
batch_size_scheduling = 
validation_criteria = loss
validation_criteria_rule = min
early_stopping_patience = 5
start_eval = False

[distortion]
p_reverb = 0.0
p_bandwidth_ext = 0.0
p_clipping = 0.0
p_air_absorption = 0.0

[optim]
lr = 0.0005
momentum = 0
weight_decay = 0.05
optimizer = adamw
lr_min = 1e-06
lr_warmup = 0.0001
warmup_epochs = 3
lr_cycle_mul = 1.0
lr_cycle_decay = 0.5
lr_cycle_epochs = -1
weight_decay_end = -1

[maskloss]
factor = 0
mask = iam
gamma = 0.6
gamma_pred = 0.6
f_under = 2

[spectralloss]
factor_magnitude = 0
factor_complex = 0
factor_under = 1
gamma = 1

[multiresspecloss]
factor = 0
factor_complex = 0
gamma = 1
fft_sizes = 512,1024,2048

[sdrloss]
factor = 0

[localsnrloss]
factor = 0.0005

and put it in a directory .../log_folder.

I also created a dataset.cfg file with the following contents:

dataset.cfg

{
  "train": [
    [
      "TRAIN_SET_SPEECH.hdf5",
      1.0
    ],
    [
      "TRAIN_SET_NOISE.hdf5",
      1.0
    ]
  ],
  "valid": [
    [
      "VALID_SET_SPEECH.hdf5",
      1.0
    ],
    [
      "VALID_SET_NOISE.hdf5",
      1.0
    ]
  ],
  "test": [
    [
      "TEST_SET_SPEECH.hdf5",
      1.0
    ],
    [
      "TEST_SET_NOISE.hdf5",
      1.0
    ]
  ]
}

Lastly, I ran the command to train the model:

python df/train.py path/to/dataset.cfg .../data_folder .../log_folder

and this is the output that I get:

2022-11-22 22:10:07 | INFO     | DF | Running on torch 1.13.0+cu117
2022-11-22 22:10:07 | INFO     | DF | Running on host workstation3
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
2022-11-22 22:10:07 | INFO     | DF | Loading model settings of log_folder
2022-11-22 22:10:07 | INFO     | DF | Running on device cuda:1
2022-11-22 22:10:07 | INFO     | DF | Initializing model `deepfilternet2`
2022-11-22 22:10:08 | DEPRECATED | DF | Use of `linear` for `df_ouput_layer` is marked as deprecated.
2022-11-22 22:10:08 | INFO     | DF | Initializing dataloader with data directory /home/tester/bokleong/dl_project_dfn/data_folder/
2022-11-22 22:10:08 | INFO     | DF | Loading HDF5 key cache from /home/tester/bokleong/dl_project_dfn/.cache_dataset.cfg
2022-11-22 22:10:08 | INFO     | DF | Start train epoch 0 with batch size 32
thread 'DataLoader Worker 1' panicked at 'called `Option::unwrap()` on a `None` value', libDF/src/dataset.rs:1134:69
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'DataLoader Worker 0' panicked at 'called `Option::unwrap()` on a `None` value', libDF/src/dataset.rs:1134:69
thread 'DataLoader Worker 2' panicked at 'called `Option::unwrap()` on a `None` value', libDF/src/dataset.rs:1134:69
thread 'DataLoader Worker 3' panicked at 'called `Option::unwrap()` on a `None` value', libDF/src/dataset.rs:1134:69
Exception in thread PinMemoryLoop:
Traceback (most recent call last):
  File "/home/tester/miniconda3/envs/user5/lib/python3.9/threading.py", line 980, in _bootstrap_inner
    self.run()
  File "/home/tester/miniconda3/envs/user5/lib/python3.9/threading.py", line 917, in run
    self._target(*self._args, **self._kwargs)
  File "/home/tester/miniconda3/envs/user5/lib/python3.9/site-packages/torch/utils/data/_utils/pin_memory.py", line 49, in _pin_memory_loop
    do_one_step()
  File "/home/tester/miniconda3/envs/user5/lib/python3.9/site-packages/torch/utils/data/_utils/pin_memory.py", line 26, in do_one_step
    r = in_queue.get(timeout=MP_STATUS_CHECK_INTERVAL)
  File "/home/tester/miniconda3/envs/user5/lib/python3.9/site-packages/libdfdata/torch_dataloader.py", line 206, in get
    self.loader.cleanup()
RuntimeError: DF dataloader error: ThreadJoinError("Any { .. }")

Appreciate any help on this, thank you!

opened by AnonymousEliforp 13

Unable to run deepFIlter form CLI

Hi guys, I have followed the installation instruction using pip and now I'm stuck with SLURM and I don't know how to fix it

I'm executing deepfilternet from colab pro+ account.

Here is the issue.

Traceback (most recent call last): File "/usr/local/bin/deepFilter", line 5, in from df.enhance import run File "/usr/local/lib/python3.7/dist-packages/df/enhance.py", line 18, in from df.logger import init_logger, warn_once File "/usr/local/lib/python3.7/dist-packages/df/logger.py", line 49 if (jobid := os.getenv("SLURM_JOB_ID")) is not None: ^ SyntaxError: invalid syntax

This is which I get while trying to execute this :-> !deepFilter /content/test_audio_053830.wav --output-dir /content

Anyone got into this kind of issue??

DO let me know the solution / how can I run this.

opened by Anand195 13

Converter to .onnx model

Hi, I've been trying to use export.py function to convert retrained model to single onnx file, but it seems that there are version mismatch of torch and onnx operation. After some modifications I've almost made it work, but now I'm stuck with

Traceback (most recent call last):
  File "/data/code_jb/deepfilter2_git/DeepFilterNet/df/export.py", line 327, in <module>
    main(args)
  File "/data/code_jb/deepfilter2_git/DeepFilterNet/df/export.py", line 292, in main
    export(
  File "/home/i.beskrovnyy/miniconda3/envs/df/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context                                                                                                                        
    return func(*args, **kwargs)
  File "/data/code_jb/deepfilter2_git/DeepFilterNet/df/export.py", line 160, in export
    export_impl(
  File "/data/code_jb/deepfilter2_git/DeepFilterNet/df/export.py", line 100, in export_impl
    torch.onnx.export(
  File "/home/i.beskrovnyy/miniconda3/envs/df/lib/python3.10/site-packages/torch/onnx/__init__.py", line 350, in export
    return utils.export(
  File "/home/i.beskrovnyy/miniconda3/envs/df/lib/python3.10/site-packages/torch/onnx/utils.py", line 163, in export
    _export(
  File "/home/i.beskrovnyy/miniconda3/envs/df/lib/python3.10/site-packages/torch/onnx/utils.py", line 1074, in _export
    graph, params_dict, torch_out = _model_to_graph(
  File "/home/i.beskrovnyy/miniconda3/envs/df/lib/python3.10/site-packages/torch/onnx/utils.py", line 731, in _model_to_graph
    graph = _optimize_graph(
  File "/home/i.beskrovnyy/miniconda3/envs/df/lib/python3.10/site-packages/torch/onnx/utils.py", line 308, in _optimize_graph
    graph = _C._jit_pass_onnx(graph, operator_export_type)
  File "/home/i.beskrovnyy/miniconda3/envs/df/lib/python3.10/site-packages/torch/onnx/__init__.py", line 416, in _run_symbolic_function                                                                                                                      
    return utils._run_symbolic_function(*args, **kwargs)
  File "/home/i.beskrovnyy/miniconda3/envs/df/lib/python3.10/site-packages/torch/onnx/utils.py", line 1421, in _run_symbolic_function                                                                                                                        
    raise symbolic_registry.UnsupportedOperatorError(
torch.onnx.symbolic_registry.UnsupportedOperatorError: Exporting the operator ::view_as_complex to ONNX opset version 14 is not supported. Please feel free to request support or submit a pull request on PyTorch GitHub.

Am I doing something wrong or the latest Deepfilternet2 modifications was not tested with this converter? Or maybe I should use specific versions of onnxruntime/torch?

opened by JBloodless 12

Noise of typing not working well

Hi Rikorose,

Thanks for working on version 2 of Deepfilternet. Now I can do the real-time inference process with buffer size=1, which is the same as the full signal effect. The point is that the state of the RNN needs to be inherited.

Now I'm having a trouble with typing/keyboard noise not working well. But I only use spectral loss with c=0.3 in Deepfilternet2 now, will multi-resolution loss improve in this case? or maybe c=0.6 in preious work is better?

Thanks, Aaron

opened by aaronhsueh0506 12
Implement a real-time loop for DeepFilterNet

Hi Hendrik, Just curious as didn't see any benchmarks but could it run chunking audio as it goes or does it need the overall file to analyse? How does it compare to https://github.com/breizhn/DTLN?

Thanks Stuart
enhancement pinned

opened by StuartIanNaylor 10
All Cargo Tests pass, but train.py returns error
I have been struggling with the same error for a while now...

I am using a conda environment in linux on a HPC solution. I previously had problems with Rust detecting multiple versions of HDF5 and (I think) I fixed it by defining HDF5_VERSION being the same as library and header (version 1.10.6) and HDF5_DIR as conda environment root. Following the instructions on https://github.com/aldanor/hdf5-rust, I also do

$ conda env config vars set RUSTFLAGS="-C link-args=-Wl,-rpath,$HDF5_DIR/lib"

And add the directory to path: $ conda develop /zhome/a7/0/155527/Desktop/s204161/DEEPL_PLS/DeepFilterNet-main/DeepFilterNet/

Doing this, I manage to succesfully pass the cargo tests (It no longer complains about HDF5 version used by rust not being the same as the one in h5py):

(DEEPL_PLS) cargo test Finished test [optimized + debuginfo] target(s) in 2.06s Running unittests src/lib.rs (target/debug/deps/libdfdata-beeffcf7e03d4848)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Running unittests src/lib.rs (target/debug/deps/libdf-9cd65e9fa81eadda)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Running unittests src/lib.rs (target/debug/deps/deep_filter_ladspa-c16148d79da4c5bf)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Running unittests src/lib.rs (target/debug/deps/df-f01c984070e59647)

running 30 tests test reexport_dataset_modules::dataset::tests::test_hdf5_slice::case_06 - should panic ... ok test reexport_dataset_modules::dataset::tests::test_hdf5_slice::case_05 - should panic ... ok test tests::test_erb_inout ... ok test reexport_dataset_modules::dataset::tests::test_mix_audio_signal ... ok test reexport_dataset_modules::dataset::tests::test_hdf5_slice::case_07 - should panic ... ok test transforms::tests::test_find_max_abs ... ok test reexport_dataset_modules::augmentations::tests::test_rand_resample ... ok test transforms::tests::test_stft_istft_delay ... ok test reexport_dataset_modules::augmentations::tests::test_low_pass ... ok test reexport_dataset_modules::augmentations::tests::test_clipping ... ok test reexport_dataset_modules::dataset::tests::test_hdf5_slice::case_10 ... ok test reexport_dataset_modules::dataset::tests::test_hdf5_read_pcm ... ok test reexport_dataset_modules::dataset::tests::test_hdf5_slice::case_04 - should panic ... ok test reexport_dataset_modules::dataset::tests::test_hdf5_slice::case_02 ... ok test reexport_dataset_modules::dataset::tests::test_hdf5_slice::case_01 ... ok test reexport_dataset_modules::dataset::tests::test_hdf5_slice::case_03 ... ok test reexport_dataset_modules::augmentations::tests::test_gen_noise ... ok test reexport_dataset_modules::dataset::tests::test_hdf5_slice::case_08 ... ok test reexport_dataset_modules::dataset::tests::test_hdf5_slice::case_09 ... ok test transforms::tests::test_ext_bandwidth_spectral ... ok test reexport_dataset_modules::augmentations::tests::test_reverb ... ok test reexport_dataset_modules::dataset::tests::test_fft_dataset ... ok test reexport_dataset_modules::augmentations::tests::test_filters ... ok test reexport_dataset_modules::augmentations::tests::test_compose ... ok test reexport_dataset_modules::dataset::tests::test_td_dataset ... ok test reexport_dataset_modules::dataset::tests::test_hdf5_read_flac ... ok test transforms::tests::test_estimate_bandwidth ... ok test reexport_dataset_modules::augmentations::tests::test_air_absorption ... ok test reexport_dataset_modules::dataset::tests::test_hdf5_read_vorbis ... ok test reexport_dataset_modules::dataloader::tests::test_data_loader ... ok

test result: ok. 30 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 6.37s

Doc-tests df

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

However when I cd into the repo and run for training it gives the error

python DeepFilterNet/df/train.py /work3/s204161/config.cfg /work3/s204161/formatted_data/ /zhome/a7/0/155527/Desktop/s204161/DEEPL_PLS/DeepFilterNet/ --debug 2022-11-21 05:42:10 | INFO | df.logger:init_logger:44 | Running on torch 1.12.0 2022-11-21 05:42:10 | INFO | df.logger:init_logger:45 | Running on host n-62-27-19 fatal: not a git repository (or any parent up to mount point /) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). 2022-11-21 05:42:10 | INFO | df.logger:init_logger:67 | Loading model settings of DeepFilterNet 2022-11-21 05:42:10 | INFO | main:main:94 | Running on device cpu 2022-11-21 05:42:10 | INFO | df.model:init_model:21 | Initializing model deepfilternet3 2022-11-21 05:42:10 | INFO | libdfdata.torch_dataloader:init:99 | Initializing dataloader with data directory /work3/s204161/formatted_data/ 2022-11-21 05:42:10 | ERROR | main::633 | An error has been caught in function '', process 'MainProcess' (16810), thread 'MainThread' (139678983296832): Traceback (most recent call last):

File "/zhome/a7/0/155527/Desktop/s204161/DEEPL_PLS/DeepFilterNet-main/DeepFilterNet/df/train.py", line 633, in main() └ <function main at 0x7f08f2c37b50>

File "/zhome/a7/0/155527/Desktop/s204161/DEEPL_PLS/DeepFilterNet-main/DeepFilterNet/df/train.py", line 139, in main dataloader = DataLoader( └ <class 'libdfdata.torch_dataloader.PytorchDataLoader'>

File "/zhome/a7/0/155527/ENTER/envs/DEEPL_PLS/lib/python3.10/site-packages/libdfdata/torch_dataloader.py", line 101, in init self.loader = _FdDataLoader( │ └ <class 'builtins._FdDataLoader'> └ <libdfdata.torch_dataloader.PytorchDataLoader object at 0x7f08f1dc6b60>

RuntimeError: DF dataset error: Hdf5ErrorDetail { source: H5Fopen(): unable to open file: bad superblock version number, msg: "Error during File::open of dataset /work3/s204161/formatted_data/VALID_SET_SPEECH.hdf5" }

I have no idea what is left to do. I do not have any prior rust experience. I've attached conda_info.txt should it be relevant and when I run the script:

import h5py hdfFile = h5py.File('/work3/s204161/formatted_data/TRAIN_SET_SPEECH.hdf5', 'r') print(h5py.version.info) for name in hdfFile: print(hdfFile[name]) print(hdfFile.keys())

Summary of the h5py configuration

h5py 3.7.0 HDF5 1.10.6 Python 3.10.8 (main, Nov 4 2022, 13:48:29) [GCC 11.2.0] sys.platform linux sys.maxsize 9223372036854775807 numpy 1.21.5 cython (built with) 0.29.30 numpy (built against) 1.21.5 HDF5 (built against) 1.10.6 <HDF5 group "/speech" (1314 members)> <KeysViewHDF5 ['speech']>

:(

conda_info.txt
opened by Student204161 9
About whitenoise performance

Thanks for your awesome work! I have installed deepfilternet through pip and test some samples. The overall noise reduction effect is great, but in whitenoise scene, there are more residues in the speech, which leads to a poor subjective feelings. Have you noticed this phenomenon? I will attach the samples below. samples.zip
stale

opened by sqummy 9
What is up with the canned deepFilter?

(DeepFilterNet) C:\Users\sdr\Downloads\Playground\DeepFilterNet>deepFilter test.wav Traceback (most recent call last): File "C:\Users\sdr\anaconda3\envs\DeepFilterNet\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\sdr\anaconda3\envs\DeepFilterNet\lib\runpy.py", line 86, in run_code exec(code, run_globals) File "C:\Users\sdr\anaconda3\envs\DeepFilterNet\Scripts\deepFilter.exe_main.py", line 7, in TypeError: main() missing 1 required positional argument: 'args'

(DeepFilterNet) C:\Users\sdr\Downloads\Playground\DeepFilterNet>
stale

opened by Joeweav 9
build(deps): bump actions/stale from 6 to 7
Bumps actions/stale from 6 to 7.

Release notes

Sourced from actions/stale's releases.

v7.0.0

⚠️ This version contains breaking changes ⚠️

What's Changed

Allow daysBeforeStale options to be float by @irega in actions/stale#841

Use cache in check-dist.yml by @jongwooo in actions/stale#876

fix print outputs step in existing workflows by @irega in actions/stale#859

Update issue and PR templates, add/delete workflow files by @IvanZosimov in actions/stale#880

Update how stale handles exempt items by @johnsudol in actions/stale#874

Breaking Changes

In this release we prevent this action from managing the stale label on items included in exempt-issue-labels and exempt-pr-labels

We decided that this is outside of the scope of this action, and to be left up to the maintainer

New Contributors

@irega made their first contribution in actions/stale#841

@jongwooo made their first contribution in actions/stale#876

@IvanZosimov made their first contribution in actions/stale#880

@johnsudol made their first contribution in actions/stale#874

Full Changelog: https://github.com/actions/stale/compare/v6...v7.0.0

v6.0.1

Update @actions/core to 1.10.0 #839

Full Changelog: https://github.com/actions/stale/compare/v6.0.0...v6.0.1

Changelog

Sourced from actions/stale's changelog.

Changelog

[7.0.0]

:warning: Breaking change :warning:

Allow daysBeforeStale options to be float by @irega in actions/stale#841

Use cache in check-dist.yml by @jongwooo in actions/stale#876

fix print outputs step in existing workflows by @irega in actions/stale#859

Update issue and PR templates, add/delete workflow files by @IvanZosimov in actions/stale#880

Update how stale handles exempt items by @johnsudol in actions/stale#874

[6.0.1]

Update @actions/core to v1.10.0 (#839)

[6.0.0]

:warning: Breaking change :warning:

Issues/PRs default close-issue-reason is now not_planned(#789)

[5.1.0]

Don't process stale issues right after they're marked stale [Add close-issue-reason option]#764 #772 Various dependabot/dependency updates

4.1.0 (2021-07-14)

Features

Ability to exempt draft PRs

4.0.0 (2021-07-14)

Features

options: simplify config by removing skip stale message options (#457) (6ec637d), closes #405 #455

output: print output parameters (#458) (3e6d35b)

Bug Fixes

dry-run: forbid mutations in dry-run (#500) (f1017f3), closes #499

logs: coloured logs (#465) (5fbbfba)

operations: fail fast the current batch to respect the operations limit (#474) (5f6f311), closes #466

label comparison: make label comparison case insensitive #517, closes #516

filtering comments by actor could have strange behavior: "stale" comments are now detected based on if the message is the stale message not who made the comment(#519), fixes #441, #509, #518

Breaking Changes

... (truncated)

Commits

6f05e42 draft release for v7.0.0 (#888)

eed91cb Update how stale handles exempt items (#874)

10dc265 Merge pull request #880 from akv-platform/update-stale-repo

9c1eb3f Update .md files and allign build-test.yml with the current test.yml

bc357bd Update .github/workflows/release-new-action-version.yml

690ede5 Update .github/ISSUE_TEMPLATE/bug_report.md

afbcabf Merge branch 'main' into update-stale-repo

e364411 Update name of codeql.yml file

627cef3 fix print outputs step (#859)

975308f Merge pull request #876 from jongwooo/chore/use-cache-in-check-dist

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependencies github_actions
opened by dependabot[bot] 0
When I retrain the model by models/DeepFilterNet2/config.ini, the result is strange~
Thanks for your awesome work! I'm trying to recurrence the training through models/DeepFilterNet2/config.ini, but when I use DeepFilterNet/df/enhance.py to test the retrained model, output wavs is nearly to silence. This is my retraining step:Thanks for your awesome work! I'm trying to recurrence the training through models/DeepFilterNet2/config.ini, but when I use DeepFilterNet/df/enhance.py to test the retrained model, output wavs is nearly to silence. This is my retraining step: (1) copy the config.ini of DeepFilterNet2

cd DeepFilterNet mkdir base_dir cp ../models/DeepFilterNet2/config.ini ./base_dir

(2) preparing data take VocalSet_48kHz_mono_000_NA_NA.tar.bz2 as clean data take noise_fullband/datasets_fullband.noise_fullband.audioset_000.tar.bz2 as noise data

find ../scripts/out/datasets_fullband/clean_fullband/VocalSet_48kHz_mono -iname "*.wav" > training_set_speech.txt python df/scripts/prepare_data.py --sr 48000 speech training_set_speech.txt ./data_dir/TRAIN_SET_SPEECH.hdf5 find ../scripts/out/datasets_fullband/noise_fullband -iname "*.wav" > training_set_noise.txt python df/scripts/prepare_data.py --sr 48000 noise training_set_noise.txt ./data_dir/TRAIN_SET_NOISE.hdf5

this is my dataset.cfg： { "train": [ [ "TRAIN_SET_SPEECH.hdf5", 1.0 ], [ "TRAIN_SET_NOISE.hdf5", 1.0 ], [ "TRAIN_SET_RIR.hdf5", 0.0 ] ], "valid": [ [ "TRAIN_SET_SPEECH.hdf5", 0.2 ], [ "TRAIN_SET_NOISE.hdf5", 0.2 ], [ "TRAIN_SET_RIR.hdf5", 0.0 ] ], "test": [ [ "TRAIN_SET_SPEECH.hdf5", 0.2 ], [ "TRAIN_SET_NOISE.hdf5", 0.2 ], [ "TRAIN_SET_RIR.hdf5", 0.0 ] ] }

(3) start retraining python df/train.py dataset.cfg ./data_dir/ ./base_dir/

(4) test retrained model

python DeepFilterNet/df/enhance.py -m DeepFilterNet/base_dir2 --output-dir DeepFilterNet/test_dir_out/ DeepFilterNet/test_dir_48k/interview_48k.wav

but the output of the reatrained model is just like this

I’m not sure which step is wrong or missed, have you ever met the same problem?
opened by Yancongs 6

Multithread ?

Hi and thanks for a amazing work ! I tested the ladspa filter on a Allwinner H3 (ARM Cortex-A7 Quad-Core 1.296 GHz) , it was easy to install; great work The filter seems to be single-threaded, a single core seems not be enough to process. Would it be possible to involve the other cores ?

2022-12-22T16:33:37.579Z | WARN |  deep_filter_ladspa | DF 7b7326a9e80d | Underrun detected (RTF: 2.39). Processing too slow!
overrun!!! (at least 277.761 ms long)
2022-12-22T16:33:38.903Z | WARN |  deep_filter_ladspa | DF 3bfb6693797b | Underrun detected (RTF: 2.48). Processing too slow!
overrun!!! (at least 312.537 ms long)
2022-12-22T16:33:40.179Z | WARN |  deep_filter_ladspa | DF 9f61b96d4beb | Underrun detected (RTF: 2.36). Processing too slow!
overrun!!! (at least 266.241 ms long)
2022-12-22T16:33:41.476Z | WARN |  deep_filter_ladspa | DF 54a1822b409f | Underrun detected (RTF: 2.41). Processing too slow!
overrun!!! (at least 284.917 ms long)

wontfix

opened by jenskastensson 2

build(deps): update roots requirement from 0.0.7 to 0.0.8 in /libDF
Updates the requirements on roots to permit the latest version.

Release notes

Sourced from roots's releases.

December 2022

Remove debug output, fix quartic equations, allow mutable closures.

Changelog

Sourced from roots's changelog.

0.0.8 - 2022-12-21

Remove debug print - thanks to J-F-Liu, Ralith

Fixed discriminant check in quartic equations - thanks to jingnanshi

Allowed mutable closures for secant etc. - thanks to vron

0.0.7 - 2021-06-17

SearchError becomes public - thanks to JP-Ellis

Unnormalized cubic equations are solved using the general formula rather than trigonometrically - thanks to Logicalshift

0.0.6 - 2019-12-22

Fixed cubic equations with very small a3 - thanks to Andrew Hunter

Improved quartic equations with multiple roots (for f64; f32 is still a problem) - thanks to Tim Lueke

Removed warnings of rustc 1.40.0

Switched benchmarks from Bencher to Criterion

0.0.5 - 2019-01-20

Trait Error implemented for SearchError - thanks to phillyfan1138

Find roots of higher-degree polynomials using eigenvalues - thanks to stiv-yakovenko

Find roots of higher-degree polynomials using Sturm's theorem recursively (experimental)

Inverse quadratic approximation

0.0.4 - 2017-09-05

Reduced the performance overhead by using generics - thanks to aepsil0n

Handle special cases of quadratic equations - thanks to stiv-yakovenko

0.0.3 - 2017-03-28

New version of the compiler

Benchmarks

Improved the speed of the Brent-Dekker method

Reduced the convergency boilerplate

0.0.2 - 2015-06-08

Fight against the compiler

[0.0.1] - 2015-03-24

Initial version

Commits

3caf974 Discontinued travis-ci.org.

28b2f21 Update changelog and version.

c4a03e1 Enable mutable closures.

cb13f4f Remove stray dbgs

8a8fed4 fix no root condition check

ad6ec95 Remove dbg macros in find_roots_quartic

See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependencies rust
opened by dependabot[bot] 0

Run tract model in half float

running

cargo run -p deep_filter --features=tract,bin,wav-utils,transforms --bin deep-filter -- --half-floats assets/noisy_snr0.wav -o out -vv -D

results in the following stack trace:

   Compiling deep_filter v0.4.1-pre (/home/hendrik/projects/DeepFilterNet/libDF)
    Finished dev [optimized + debuginfo] target(s) in 20.20s
     Running `target/debug/deep-filter --half-floats assets/noisy_snr0.wav -o out -vv -D`
[2022-12-13T10:09:59Z INFO  df::tract] Init encoder with delay: 0
[2022-12-13T10:09:59Z INFO  tract_linalg::x86_64_fma] mmm_f32, sigmoid_f32, tanh_f32: x86_64/fma activated
[2022-12-13T10:09:59Z INFO  tract_linalg::x86_64_fma] mmm_i8_i8 and mmm_i8_i32: x86_64/avx2 activated
[2022-12-13T10:09:59Z INFO  df::tract] Init ERB decoder with delay: 0
[2022-12-13T10:09:59Z INFO  df::tract] Init DF decoder with delay: 0
[2022-12-13T10:09:59Z INFO  df::tract] Running with model type deepfilternet3 lookahead 2
Error: Evaluating #3 "/erb_conv0/1/Conv.pad" Pad

Caused by:
    Tensor datum type error: tensor is F32, accessed as F16

Stack backtrace:
   0: tract_data::tensor::Tensor::check_for_access
             at /home/hendrik/.cargo/registry/src/github.com-1ecc6299db9ec823/tract-data-0.18.5/src/tensor.rs:748:13
   1: tract_data::tensor::Tensor::to_scalar
             at /home/hendrik/.cargo/registry/src/github.com-1ecc6299db9ec823/tract-data-0.18.5/src/tensor.rs:848:9
   2: tract_core::ops::array::pad::Pad::eval_t
             at /home/hendrik/.cargo/registry/src/github.com-1ecc6299db9ec823/tract-core-0.18.5/src/ops/array/pad.rs:34:38
      <tract_core::ops::array::pad::Pad as tract_core::ops::EvalOp>::eval
             at /home/hendrik/.cargo/registry/src/github.com-1ecc6299db9ec823/tract-core-0.18.5/src/ops/array/pad.rs:109:18
   3: tract_core::plan::eval
             at /home/hendrik/.cargo/registry/src/github.com-1ecc6299db9ec823/tract-core-0.18.5/src/plan.rs:491:17
   4: core::ops::function::FnMut::call_mut
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/core/src/ops/function.rs:164:5
      tract_core::plan::SimpleState<F,O,M,P>::exec_plan_with_eval
             at /home/hendrik/.cargo/registry/src/github.com-1ecc6299db9ec823/tract-core-0.18.5/src/plan.rs:264:26
   5: tract_core::plan::SimpleState<F,O,M,P>::run_plan_with_eval
             at /home/hendrik/.cargo/registry/src/github.com-1ecc6299db9ec823/tract-core-0.18.5/src/plan.rs:199:9
      tract_core::plan::SimpleState<F,O,M,P>::run
             at /home/hendrik/.cargo/registry/src/github.com-1ecc6299db9ec823/tract-core-0.18.5/src/plan.rs:177:9
   6: df::tract::DfTract::process
             at ./libDF/src/tract.rs:401:17
   7: deep_filter::main
             at ./libDF/src/bin/enhance_wav.rs:139:13
   8: core::ops::function::FnOnce::call_once
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/core/src/ops/function.rs:248:5
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/sys_common/backtrace.rs:122:18
   9: std::rt::lang_start::{{closure}}
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/rt.rs:166:18
  10: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/core/src/ops/function.rs:283:13
      std::panicking::try::do_call
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/panicking.rs:492:40
      std::panicking::try
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/panicking.rs:456:19
      std::panic::catch_unwind
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/panicking.rs:492:40
      std::panicking::try
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/panicking.rs:456:19
      std::panic::catch_unwind
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/rt.rs:148:20
  11: main
  12: __libc_start_call_main
  13: __libc_start_main_alias_2
  14: _start

opened by Rikorose 2