Python library for handling audio datasets.

Last update: Nov 27, 2022

Overview

AUDIOMATE

Audiomate is a library for easy access to audio datasets. It provides the datastructures for accessing/loading different datasets in a generic way. This should ease the use of audio datasets for example for machine learning tasks.

import audiomate
from audiomate.corpus import io

# Download a dataset
esc_downloader = io.ESC50Downloader()
esc_downloader.download('/local/path')

# Load and work with the dataset
esc50 = audiomate.Corpus.load('/local/path', reader='esc-50')

# e.g. Read the audio signal and the label of specific sample/utterance
utterance = esc50.utterances['1-100032-A-0']
samples = utterance.read_samples()
label_list = utterance.label_lists[audiomate.corpus.LL_SOUND_CLASS]

for label in label_list:
  print(label.start, label.value)

Furthermore it provides tools for interacting with datasets (validation, splitting, subsets, merge, filter), extracting features, feeding samples for training ML models and more.

Currently supported datasets:

Currently supported formats:

Installation

pip install audiomate

Install the latest development version:

pip install git+https://github.com/ynop/audiomate.git

Dependencies

sox

For parts of the functionality (e.g. audio format conversion) sox is used. In order to use it, you have to install sox.

# macos
brew install sox

# with support for specific formats
brew install sox --with-lame --with-flac --with-libvorbis

# linux
apt-get install sox

# anaconda for macOS/windows/linux:
conda install -c conda-forge sox

Development

Prerequisites

A supported version of Python > 3.5

It's recommended to use a virtual environment when developing audiomate. To create one, execute the following command in the project's root directory:

python -m venv .

To install audiomate and all it's dependencies, execute:

pip install -e .

Running the test suite

pip install -e .[dev]
pytest

With PyCharm you might have to change the default test runner. Otherwise, it might only suggest to use nose. To do so, go to File > Settings > Tools > Python Integrated Tools (on the Mac it's PyCharm > Preferences > Settings > Tools > Python Integrated Tools) and change the test runner to py.test.

Benchmarks

In order to check the runtime of specific parts, pytest-benchmark is used. Benchmarks are normal test functions, but call the benchmark fixture for the code under test.

To run benchmarks:

# Run all
pytest bench

# Specific benchmark
pytest bench/corpus/test_merge_corpus.py

To compare between different runs:

pytest-benchmark compare

Editing the Documentation

The documentation is written in reStructuredText and transformed into various output formats with the help of Sphinx.

To generate the documentation, execute:

pip install -e .[dev]
cd docs
make html

The generated files are written to docs/_build/html.

Versions

Versions is handled using bump2version. To bump the version:

bump2version [major,minor,patch,release,num]

In order to directly go to a final relase version (skip .dev/.rc/...):

bump2version [major,minor,patch] --new-version x.x.x

Release

Commands to create a new release on pypi.

rm -rf build
rm -rf dist

python setup.py sdist
python setup.py bdist_wheel
twine upload dist/*

Comments

Downloader for the Mozilla Common Voice Datasets
This 3 commits add support for downloading the Common Voice datasets. All today available languages are supported. If new languages will be added, the provided tests fail or a new release of the Common Voice dataset is available then the DOWNLOAD_URLS need to be modified to include/replace the new/broken URL's.

Hopefully the commit message is OK

flake8 gives no errors

Docstring is added to the Downloader Class

Used ArchivDownloader as Base so Downloader is tested -> ~~Dataset availability tests (checks if HTTP HEAD Status Code is 200) are added~~

Is straight with master
opened by Mastercuber 10
Show tatoeba process

I also changed the label type, because this solved a problem i had and the other datasets seemed to have LL_WORD_TRANSCRIPT as default and LL_WORD_TRANSCRIPT_RAW only optionally.

opened by DanBmh 10
Corpus with new common-voice dataset released june 2020 idx out of range

Dear Support.

Can you just readjust it with the new common-voice dataset? like I tested with German and it is throwing error, idx out of range. I tried to remove first two columns and use just client_id, file_path and text. But still error exists.

Can you help?

opened by Tortoise17 6
Check if file exists before downloading - GH #125
I've added the following changes per our discussion in #125:

Added force_redownload flag as optional arg

Added check for existing path to prevent duplicate downloads of corporum (unless flag is turned on)

Updated docstring to reflect new arg and exception

pytest on "audiomate/test/corpus/io/" yields 8 failures both before and after changes. Please let me know what you think whenever you have a chance!
opened by cdkini 6

Check if files exist before downloading

Running

esc_downloader = io.ESC50Downloader()
esc_downloader.download('esc')
esc_downloader.download('esc')

downloads a dataset twice. Whereas the second time the download is completed not saved to the target folder because the folder already exists.

Download File: 647Mit [01:24, 7.65Mit/s]
Download File: 647Mit [01:35, 6.79Mit/s]
Traceback (most recent call last):
  File "test.py", line 7, in <module>
    esc_downloader.download('esc')
  File "/Users/faro/repositories/audiomate_test/env/lib/python3.6/site-packages/audiomate/corpus/io/base.py", line 28, in download
    return self._download(target_path)
  File "/Users/faro/repositories/audiomate_test/env/lib/python3.6/site-packages/audiomate/corpus/io/downloader.py", line 52, in _download
    delete_subfolders=True
  File "/Users/faro/repositories/audiomate_test/env/lib/python3.6/site-packages/audiomate/utils/files.py", line 29, in move_all_files_from_subfolders_to_top
    shutil.move(src, target)
  File "/Users/faro/repositories/audiomate_test/env/lib/python3.6/shutil.py", line 548, in move
    raise Error("Destination path '%s' already exists" % real_dst)
shutil.Error: Destination path 'esc/tests/tests' already exists

either check the existence of a root path before downloading any files at least raise a warning.
or add a force_redownload flag that would forcefully redownload the dataset (even if already existing)

this issue is part of a JOSS review

enhancement good first issue

opened by faroit 6

Catch file conversion error and add a lot of invalid files.

Note: I did not check these automatically excluded files. They threw an error in my deepspeech training. But earlier i did check some of the tuda files throwing the error and they all had a wrong transcription.

opened by DanBmh 5

Unit test error with `TestLabel.test_read_samples`

I am getting the following error with the unit tests:

_________________________ TestLabel.test_read_samples __________________________

self = <tests.annotations.test_label.TestLabel object at 0x7f800dfdb7f0>

    def test_read_samples(self):
        path = resources.sample_wav_file('wav_1.wav')
        track = tracks.FileTrack('wav', path)
        issuer = issuers.Issuer('toni')
        utt = tracks.Utterance('t', track, issuer=issuer, start=1.0, end=2.30)
    
        l1 = annotations.Label('a', 0.15, 0.448)
        l2 = annotations.Label('a', 0.5, 0.73)
        ll = annotations.LabelList(labels=[l1, l2])
    
        utt.set_label_list(ll)
    
        expected, __ = librosa.core.load(
            path,
            sr=None,
            offset=1.15,
            duration=0.298
        )
        assert np.array_equal(l1.read_samples(), expected)
    
        expected, __ = librosa.core.load(
            path,
            sr=None,
            offset=1.5,
            duration=0.23
        )
>       assert np.array_equal(l2.read_samples(), expected)
E       assert False
E        +  where False = <function array_equal at 0x7f803bdbe598>(array([-0.00274658, -0.00021362, -0.00115967, ..., -0.00204468,\n       -0.00167847,  0.00067139], dtype=float32), array([-0.00274658, -0.00021362, -0.00115967, ..., -0.00167847,\n        0.00067139,  0.00082397], dtype=float32))
E        +    where <function array_equal at 0x7f803bdbe598> = np.array_equal
E        +    and   array([-0.00274658, -0.00021362, -0.00115967, ..., -0.00204468,\n       -0.00167847,  0.00067139], dtype=float32) = <bound method Label.read_samples of Label(a, 0.5, 0.73)>()
E        +      where <bound method Label.read_samples of Label(a, 0.5, 0.73)> = Label(a, 0.5, 0.73).read_samples

I am not sure what the issue is.

opened by toddrme2178 4

New datasets and improved features
This adds following new datasets:

Css10

LinguaLibre

Zamia-Speech

The common voice reader now can import the new v5 release and has more languages.

Some dataset loading problems were fixed.

I also did add the progress logging bar again, because I missed this feature (but through audiomates logging interface that the used library can be easily exchanged)

Note: This also changes the license to LGPL and I did use black for formatting
opened by DanBmh 3

Test failure in `test_split_by_length_of_utterances`

I am getting the following unit test error:

______________________ test_split_by_length_of_utterances ______________________

benchmark = <pytest_benchmark.fixture.BenchmarkFixture object at 0x7f076e177a58>

    def test_split_by_length_of_utterances(benchmark):
        corpus = resources.generate_corpus(
            179,
            (250, 500),
            (1, 9),
            (0, 6),
            (1, 20),
            random.Random(x=234)
        )
    
        splitter = subset.Splitter(corpus, random_seed=324)
    
>       benchmark(run, splitter)

bench/corpus/subset/splitting/test_split_by_length_of_utterances.py:33: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib/python3.7/site-packages/pytest_benchmark/fixture.py:126: in __call__
    return self._raw(function_to_benchmark, *args, **kwargs)
/usr/lib/python3.7/site-packages/pytest_benchmark/fixture.py:148: in _raw
    duration, iterations, loops_range = self._calibrate_timer(runner)
/usr/lib/python3.7/site-packages/pytest_benchmark/fixture.py:276: in _calibrate_timer
    duration = runner(loops_range)
/usr/lib/python3.7/site-packages/pytest_benchmark/fixture.py:91: in runner
    function_to_benchmark(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

splitter = <audiomate.corpus.subset.splitting.Splitter object at 0x7f076e177e80>

    def run(splitter):
>       out = splitter.split_by_length_of_utterances(
            {
                'train': 0.7,
                'test': 0.15,
                'dev': 0.15
            },
            separate_issuers=True
        )
E       AttributeError: 'Splitter' object has no attribute 'split_by_length_of_utterances'

bench/corpus/subset/splitting/test_split_by_length_of_utterances.py:9: AttributeError

It looks like the method split_by_length_of_utterances has been renamed. Having CI tests running, such as travis, might help with this sort of issue.

opened by toddrme2178 3

Updating license

I'd like to add some new datasets and improved features (see https://github.com/ynop/audiomate/pull/129), but I'd also like to update the licence for this (MIT->LGPL).

I'm developing all of this in my free time and want to make sure that everyone who uses and extends it also contributes back again, that it becomes an exchange of work and time. I also had some bad experiences with a partly open source and partly proprietary project where I put a lot of time into and which I can't use any more because the community support was stopped and development is now continued proprietary. That's why I generally prefer (weak) copyleft licenses. (Not that I believe this will happen here.)

Using LGPL shouldn't affect any users, so from my point of view it has no negative consequences except the effort of the changing process, but ensures the exchange of work.

opened by DanBmh 2
Kaldi I/O type error

Dear friend.

as I suppose, the audiomate is handling lot of formats very efficiently and perfectly.

but, there is one format issue. Maybe you can understand. while data preparation, it throws error below

File "/home/user/anaconda3/envs/zml/lib/python3.6/site-packages/audiomate/corpus/corpus.py", line 123, in load return reader.load(path) File "/home/user/anaconda3/envs/zml/lib/python3.6/site-packages/audiomate/corpus/io/base.py", line 95, in load self.type(), ' '.join(missing_files), path))

OSError: Invalid data set of type kaldi: files wav.scp text not found at data/swc

The data in each folder is consisting files below.
aligned.swc audio.ogg audiometa.txt info.json wiki.html wiki.html wiki.xml

Do I need to run any script first to convert these formats? or ?any of the library is ignored by me to install. Can you comment? why is it so? or any script to resolve and make like to be useable for audiomate.

opened by Tortoise17 2
CVE-2007-4559 Patch

Patching CVE-2007-4559

Hi, we are security researchers from the Advanced Research Center at Trellix. We have began a campaign to patch a widespread bug named CVE-2007-4559. CVE-2007-4559 is a 15 year old bug in the Python tarfile package. By using extract() or extractall() on a tarfile object without sanitizing input, a maliciously crafted .tar file could perform a directory path traversal attack. We found at least one unsantized extractall() in your codebase and are providing a patch for you via pull request. The patch essentially checks to see if all tarfile members will be extracted safely and throws an exception otherwise. We encourage you to use this patch or your own solution to secure against CVE-2007-4559. Further technical information about the vulnerability can be found in this blog.

If you have further questions you may contact us through this projects lead researcher Kasimir Schulz.

opened by TrellixVulnTeam 0
Bump numpy from 1.18.1 to 1.22.0
Bumps numpy from 1.18.1 to 1.22.0.

Release notes

Sourced from numpy's releases.

v1.22.0

NumPy 1.22.0 Release Notes

NumPy 1.22.0 is a big release featuring the work of 153 contributors spread over 609 pull requests. There have been many improvements, highlights are:

Annotations of the main namespace are essentially complete. Upstream is a moving target, so there will likely be further improvements, but the major work is done. This is probably the most user visible enhancement in this release.

A preliminary version of the proposed Array-API is provided. This is a step in creating a standard collection of functions that can be used across application such as CuPy and JAX.

NumPy now has a DLPack backend. DLPack provides a common interchange format for array (tensor) data.

New methods for quantile, percentile, and related functions. The new methods provide a complete set of the methods commonly found in the literature.

A new configurable allocator for use by downstream projects.

These are in addition to the ongoing work to provide SIMD support for commonly used functions, improvements to F2PY, and better documentation.

The Python versions supported in this release are 3.8-3.10, Python 3.7 has been dropped. Note that 32 bit wheels are only provided for Python 3.8 and 3.9 on Windows, all other wheels are 64 bits on account of Ubuntu, Fedora, and other Linux distributions dropping 32 bit support. All 64 bit wheels are also linked with 64 bit integer OpenBLAS, which should fix the occasional problems encountered by folks using truly huge arrays.

Expired deprecations

Deprecated numeric style dtype strings have been removed

Using the strings "Bytes0", "Datetime64", "Str0", "Uint32", and "Uint64" as a dtype will now raise a TypeError.

(gh-19539)

Expired deprecations for loads, ndfromtxt, and mafromtxt in npyio

numpy.loads was deprecated in v1.15, with the recommendation that users use pickle.loads instead. ndfromtxt and mafromtxt were both deprecated in v1.17 - users should use numpy.genfromtxt instead with the appropriate value for the usemask parameter.

(gh-19615)

... (truncated)

Commits

4adc87d Merge pull request #20685 from charris/prepare-for-1.22.0-release

fd66547 REL: Prepare for the NumPy 1.22.0 release.

125304b wip

c283859 Merge pull request #20682 from charris/backport-20416

5399c03 Merge pull request #20681 from charris/backport-20954

f9c45f8 Merge pull request #20680 from charris/backport-20663

794b36f Update armccompiler.py

d93b14e Update test_public_api.py

7662c07 Update init.py

311ab52 Update armccompiler.py

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies
opened by dependabot[bot] 0
do not call head if filesize is provided with headers

The download URL in swc.py is not valid anymore. The new server location does not support a HTTP head request. This patch is taking the file size from the previous GET and if the file size is known the head call is not executed.

opened by kaoh 0
Retry download if closed connection without response

If server close connection, you dont have a "response code" and fail into a Exception.

Proof's:

`/usr/local/lib/python3.6/dist-packages/librosa/util/decorators.py:9: NumbaDeprecationWarning: An import was requested from a module that has moved location. Import of 'jit' requested from: 'numba.decorators', please update to use 'numba.core.decorators' or pin to Numba version 0.48.0. This alias will not be present in Numba version 0.50.0. from numba.decorators import jit as optional_jit Downloading voxforge-es ... INFO:audiomate:Found 2244 available archives to download Download Files: 0%| | 0/2244 [00:00<?, ?it/s]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/ARI-20100216-hds.tgz" with size: 1361655 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AbdielNavarro-20130409-xvp.tgz" with size: 2625640 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AGOs-20100831-jdm.tgz" with size: 1915692 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AGOs-20100831-tmi.tgz" with size: 2009016 B INFO:audiomate:Finished download INFO:audiomate:Finished download INFO:audiomate:Finished download INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Adrian-20120203-lit.tgz" with size: 2607824 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AdrianRamirezEscalante-20111202-pgl.tgz" with size: 2594505 B INFO:audiomate:Finished download INFO:audiomate:Finished download Download Files: 0%| | 1/2244 [00:12<7:30:05, 12.04s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Adrin-20120203-uup.tgz" with size: 2230035 B INFO:audiomate:Finished download Download Files: 0%|▎ | 5/2244 [00:18<5:31:54, 8.89s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Aguatacuperche-20100823-zvv.tgz" with size: 2132406 B INFO:audiomate:Finished download Download Files: 0%|▍ | 7/2244 [00:21<4:11:57, 6.76s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Agus-20120528-sza.tgz" with size: 1533718 B INFO:audiomate:Finished download Download Files: 0%|▌ | 8/2244 [00:28<4:06:45, 6.62s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AlexVixgeck-20090827-qms.tgz" with size: 1105585 B INFO:audiomate:Finished download Download Files: 0%|▌ | 9/2244 [00:29<3:05:11, 4.97s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AlanNaiman-20121014-tke.tgz" with size: 2065020 B INFO:audiomate:Finished download INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alejandra-20111109-oaf.tgz" with size: 1311472 B INFO:audiomate:Finished download Download Files: 0%|▋ | 10/2244 [00:37<3:37:16, 5.84s/it]INFO:audiomate:Finished download Download Files: 0%|▋ | 11/2244 [00:38<2:45:46, 4.45s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20091127-dls.tgz" with size: 2144399 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20091127-wrq.tgz" with size: 2084240 B INFO:audiomate:Finished download INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AlexanderRubioCaceres-20090902-set.tgz" with size: 1260802 B INFO:audiomate:Finished download INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20091128-fan.tgz" with size: 2444151 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20091127-dis.tgz" with size: 2013014 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20091128-ozc.tgz" with size: 2397126 B INFO:audiomate:Finished download Download Files: 1%|▊ | 13/2244 [00:46<2:39:44, 4.30s/it]INFO:audiomate:Finished download INFO:audiomate:Finished download INFO:audiomate:Finished download Download Files: 1%|▉ | 14/2244 [00:53<3:14:10, 5.22s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20100520-svz.tgz" with size: 2348385 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20091128-ygd.tgz" with size: 2095451 B INFO:audiomate:Finished download INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20091128-zoj.tgz" with size: 2226773 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20100520-wvo.tgz" with size: 2189445 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20091128-suc.tgz" with size: 2266510 B INFO:audiomate:Finished download INFO:audiomate:Finished download INFO:audiomate:Finished download Download Files: 1%|█▏ | 19/2244 [01:06<2:43:43, 4.42s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20110915-mnf.tgz" with size: 2442328 B INFO:audiomate:Finished download Download Files: 1%|█▍ | 23/2244 [01:08<2:00:56, 3.27s/it]INFO:audiomate:Finished download Download Files: 1%|█▌ | 24/2244 [01:09<1:38:17, 2.66s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20110915-sff.tgz" with size: 2128985 B INFO:audiomate:Finished download Download Files: 1%|█▋ | 25/2244 [01:12<1:44:04, 2.81s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20110915-wfh.tgz" with size: 2021008 B INFO:audiomate:Finished download Download Files: 1%|█▋ | 26/2244 [01:20<2:35:21, 4.20s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alfredo-20090314-you.tgz" with size: 2117498 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alfredo-20090314-bwm.tgz" with size: 1780407 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alfredo-20090314-nui.tgz" with size: 2000040 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreaBurgos-20120424-pef.tgz" with size: 1929018 B INFO:audiomate:Finished download INFO:audiomate:Finished download Download Files: 1%|█▊ | 27/2244 [01:32<4:01:22, 6.53s/it]INFO:audiomate:Finished download Download Files: 1%|█▊ | 28/2244 [01:33<2:58:04, 4.82s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130708-inn.tgz" with size: 1721089 B INFO:audiomate:Finished download INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130710-dtw.tgz" with size: 1736300 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130708-eaz.tgz" with size: 1909391 B INFO:audiomate:Finished download Download Files: 1%|█▉ | 30/2244 [01:38<2:35:58, 4.23s/it]INFO:audiomate:Finished download INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130710-bhv.tgz" with size: 1776404 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130710-iqt.tgz" with size: 1986977 B INFO:audiomate:Finished download INFO:audiomate:Finished download Download Files: 1%|██ | 31/2244 [01:45<3:04:53, 5.01s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130710-ttq.tgz" with size: 1749826 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130710-stu.tgz" with size: 1753877 B INFO:audiomate:Finished download Download Files: 1%|██▏ | 33/2244 [01:49<2:27:32, 4.00s/it]INFO:audiomate:Finished download INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130710-rat.tgz" with size: 1553412 B INFO:audiomate:Finished download INFO:audiomate:Finished download Download Files: 2%|██▎ | 36/2244 [01:55<2:07:09, 3.46s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130710-twg.tgz" with size: 1853932 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130720-mkx.tgz" with size: 1973814 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130720-gix.tgz" with size: 1687520 B INFO:audiomate:Finished download INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130711-jje.tgz" with size: 1589503 B INFO:audiomate:Finished download Download Files: 2%|██▌ | 39/2244 [02:02<1:55:15, 3.14s/it]INFO:audiomate:Finished download INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AntonioH-20160531-gwl.tgz" with size: 849178 B INFO:audiomate:Finished download INFO:audiomate:Finished download Download Files: 2%|██▌ | 40/2244 [02:08<2:22:21, 3.88s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Andros-20120408-age.tgz" with size: 2205975 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Argorn-20141126-igm.tgz" with size: 2165540 B INFO:audiomate:Finished download INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Ases-20120213-esa.tgz" with size: 2552878 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130720-slz.tgz" with size: 1946756 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AntonioMedina-20121208-kzq.tgz" with size: 2329390 B INFO:audiomate:Finished download INFO:audiomate:Finished download Download Files: 2%|██▊ | 43/2244 [02:22<2:33:13, 4.18s/it]INFO:audiomate:Finished download INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Ashrey-20100714-yhd.tgz" with size: 1937711 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/BLs-20151219-kwe.tgz" with size: 2128918 B INFO:audiomate:Finished download Download Files: 2%|██▉ | 46/2244 [02:26<2:01:53, 3.33s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/BLs-20151219-tsj.tgz" with size: 1742771 B INFO:audiomate:Finished download Download Files: 2%|███▏ | 49/2244 [02:32<1:44:45, 2.86s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/BLs-20151219-vvx.tgz" with size: 2222457 B INFO:audiomate:Finished download INFO:audiomate:Finished download INFO:audiomate:Finished download INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Azul4-20131019-ssk.tgz" with size: 2093735 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Campotrigo-20130607-bnz.tgz" with size: 3093233 B INFO:audiomate:Finished download INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/CarlosEspinoAngulo-20140525-vip.tgz" with size: 2302281 B INFO:audiomate:Finished download INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/BLs-20151219-xzd.tgz" with size: 1994095 B INFO:audiomate:Finished download INFO:audiomate:Finished download Download Files: 2%|███▎ | 50/2244 [02:49<4:22:27, 7.18s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/CarlosMedina-20130524-wqx.tgz" with size: 2357299 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/CarlosFiebigCosmelli-20121128-aud.tgz" with size: 1756883 B INFO:audiomate:Finished download INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Carlos_Daniel-20120325-khj.tgz" with size: 2414791 B INFO:audiomate:Finished download INFO:audiomate:Finished download INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Condor-20120214-prz.tgz" with size: 1716730 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Bernardo-20101210-tiq.tgz" with size: 1849584 B INFO:audiomate:Finished download INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Cixert-20150207-ywe.tgz" with size: 1980326 B INFO:audiomate:Finished download Download Files: 3%|███▊ | 58/2244 [03:05<1:56:45, 3.20s/it] Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 677, in urlopen chunked=chunked, File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 426, in _make_request six.raise_from(e, None) File "", line 3, in raise_from File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 421, in _make_request httplib_response = conn.getresponse() File "/usr/lib/python3.6/http/client.py", line 1356, in getresponse response.begin() File "/usr/lib/python3.6/http/client.py", line 307, in begin version, status, reason = self._read_status() File "/usr/lib/python3.6/http/client.py", line 276, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 449, in send timeout=timeout File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 727, in urlopen method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] File "/usr/local/lib/python3.6/dist-packages/urllib3/util/retry.py", line 403, in increment raise six.reraise(type(error), error, _stacktrace) File "/usr/local/lib/python3.6/dist-packages/urllib3/packages/six.py", line 734, in reraise raise value.with_traceback(tb) File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 677, in urlopen chunked=chunked, File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 426, in _make_request six.raise_from(e, None) File "", line 3, in raise_from File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 421, in _make_request httplib_response = conn.getresponse() File "/usr/lib/python3.6/http/client.py", line 1356, in getresponse response.begin() File "/usr/lib/python3.6/http/client.py", line 307, in begin version, status, reason = self._read_status() File "/usr/lib/python3.6/http/client.py", line 276, in _read_status raise RemoteDisconnected("Remote end closed connection without" urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/DeepSpeech/deepspeech-polyglot/preprocessing/download_data.py", line 108, in main() File "/DeepSpeech/deepspeech-polyglot/preprocessing/download_data.py", line 97, in main dl.download(os.path.join(args.target_path, "voxforge")) File "/usr/local/lib/python3.6/dist-packages/audiomate/corpus/io/base.py", line 40, in download return self._download(target_path) File "/usr/local/lib/python3.6/dist-packages/audiomate/corpus/io/voxforge.py", line 73, in _download downloaded = self.download_files(available, temp_folder) File "/usr/local/lib/python3.6/dist-packages/audiomate/corpus/io/voxforge.py", line 110, in download_files dl_result = download.download_files(url_to_target, num_threads=self.num_workers) File "/usr/local/lib/python3.6/dist-packages/audiomate/utils/download.py", line 35, in download_files description='Download Files' File "/usr/local/lib/python3.6/dist-packages/audiomate/logutil.py", line 21, in progress for x in tqdm.tqdm(iterable, desc=description, total=total): File "/usr/local/lib/python3.6/dist-packages/tqdm/std.py", line 1165, in iter for obj in iterable: File "/usr/lib/python3.6/multiprocessing/pool.py", line 735, in next raise value File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker result = (True, func(*args, **kwds)) File "/usr/local/lib/python3.6/dist-packages/audiomate/utils/download.py", line 43, in _download_file return download_file(item[0], item[1]) File "/usr/local/lib/python3.6/dist-packages/audiomate/utils/download.py", line 62, in download_file r = requests.get(url, stream=True) File "/usr/local/lib/python3.6/dist-packages/requests/api.py", line 76, in get return request('get', url, params=params, **kwargs) File "/usr/local/lib/python3.6/dist-packages/requests/api.py", line 61, in request return session.request(method=method, url=url, **kwargs) File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 530, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 643, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 498, in send raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))`

opened by CostanzoPablo 0
Reproducibility of dataset downloads

When downloading corpora from versioned data stores, I would expect to take into account a tag or specific hash of that dataset. That way users are sure if a specific version of audiomate yields an identical corpus to foster reproducibility.

e.g. lets take the esc-50 corpus: the root url downloads directly from master branch https://github.com/ynop/audiomate/blob/28696c0e46ab1d7f3d3f53f8a9086c724b7b947a/audiomate/corpus/io/esc.py#L11

To improve reproducibility, I suggest that audiomate uses tags where possible (github, zenodo, ...) and furthermore provide a checksum mechanism that verifies a successful download.

This issue is part of a JOSS review https://github.com/openjournals/joss-reviews/issues/2135
enhancement

opened by faroit 2