Dataset and baseline code for the VocalSound dataset (ICASSP2022).

Overview

VocalSound: A Dataset for Improving Human Vocal Sounds Recognition

Introduction

VocalSound Poster

VocalSound is a free dataset consisting of 21,024 crowdsourced recordings of laughter, sighs, coughs, throat clearing, sneezes, and sniffs from 3,365 unique subjects. The VocalSound dataset also contains meta information such as speaker age, gender, native language, country, and health condition.

This repository contains the official code of the data preparation and baseline experiment in the ICASSP paper VocalSound: A Dataset for Improving Human Vocal Sounds Recognition (Yuan Gong, Jin Yu, and James Glass; MIT & Signify). Specifically, we provide an extremely simple one-click Google Colab script Open In Colab for the baseline experiment, no GPU / local data downloading is needed.

The dataset is ideal for:

  • Build vocal sound recognizer.
  • Research on removing model bias on various speaker groups.
  • Evaluate pretrained models (e.g., those trained with AudioSet) on vocal sound classification to check their generalization ability.
  • Combine with existing large-scale general audio dataset to improve the vocal sound recognition performance.

Citing

Please cite our paper(s) if you find the VocalSound dataset and code useful. The first paper proposes introduces the VocalSound dataset and the second paper describes the training pipeline and model we used for the baseline experiment.

@INPROCEEDINGS{gong_vocalsound,
  author={Gong, Yuan and Yu, Jin and Glass, James},
  booktitle={ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, 
  title={Vocalsound: A Dataset for Improving Human Vocal Sounds Recognition}, 
  year={2022},
  pages={151-155},
  doi={10.1109/ICASSP43922.2022.9746828}}
@ARTICLE{gong_psla, 
    author={Gong, Yuan and Chung, Yu-An and Glass, James},
    title={PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregation}, 
    journal={IEEE/ACM Transactions on Audio, Speech, and Language Processing},  
    year={2021}, 
    doi={10.1109/TASLP.2021.3120633}
}

Download VocalSound

The VocalSound dataset can be downloaded as a single .zip file:

Sample Recordings (Listen to it without downloading)

VocalSound 44.1kHz Version (4.5 GB)

VocalSound 16kHz Version (1.7 GB, used in our baseline experiment)

(Mirror Links) 腾讯微云下载链接: 试听24个样本16kHz版本44.1kHz版本

If you plan to reproduce our baseline experiments using our Google Colab script, you do NOT need to download it manually, our script will download and process the 16kHz version automatically.

Creative Commons License
The VocalSound dataset is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Dataset Details

data
├──readme.txt
├──class_labels_indices_vs.csv # include label code and name information
├──audio_16k
│  ├──f0003_0_cough.wav # female speaker, id=0003, 0=first collection (most spks only record once, but there are exceptions), cough
│  ├──f0003_0_laughter.wav
│  ├──f0003_0_sigh.wav
│  ├──f0003_0_sneeze.wav
│  ├──f0003_0_sniff.wav
│  ├──f0003_0_throatclearing.wav
│  ├──f0004_0_cough.wav # data from another female speaker 0004
│   ... (21024 files in total)
│   
├──audio_44k
│    # same recordings with those in data/data_16k, but are no downsampled
│   ├──f0003_0_cough.wav
│    ... (21024 files in total)
│
├──datafiles  # json datafiles that we use in our baseline experiment, you can ignore it if you don't use our training pipeline
│  ├──all.json  # all data
│  ├──te.json  # test data
│  ├──tr.json  # training data
│  ├──val.json  # validation data
│  └──subtest # subset of the test set, for fine-grained evaluation
│     ├──te_age1.json  # age [18-25]
│     ├──te_age2.json  # age [26-48]
│     ├──te_age3.json  # age [49-80]
│     ├──te_female.json
│     └──te_male.json
│
└──meta  # Meta information of the speakers [spk_id, gender, age, country, native language, health condition (no=no problem)]
   ├──all_meta.json  # all data
   ├──te_meta.json  # test data
   ├──tr_meta.json  # training data
   └──val_meta.json  # validation data

Baseline Experiment

Option 1. One-Click Google Colab Experiment Open In Colab

We provide an extremely simple one-click Google Colab script for the baseline experiment.

What you need:

  • A free google account with Google Drive free space > 5Gb
    • A (paid) Google Colab Pro plan could speed up training, but is not necessary. Free version can run the script, just a bit slower.

What you don't need:

  • Download VocalSound manually (The Colab script download it to your Google Drive automatically)
  • GPU or any other hardware (Google Colab provides free GPUs)
  • Any enviroment setting and package installation (Google Colab provides a ready-to-use environment)
  • A specific operating system (You only need a web browser, e.g., Chrome)

Please Note

  • This script is slightly different with our local code, but the performance is not impacted.
  • Free Google Colab might be slow and unstable. In our test, it takes ~5 minutes to train the model for one epoch with a free Colab account.

To run the baseline experiment

  • Make sure your Google Drive is mounted. You don't need to do it by yourself, but Google Colab will ask permission to acess your Google Drive when you run the script, please allow it if you want to use Google Drive.
  • Make sure GPU is enabled for Colab. To do so, go to the top menu > Edit > Notebook settings and select GPU as Hardware accelerator.
  • Run the script. Just press Ctrl+F9 or go to runtime menu on top and click "run all" option. That's it.

Option 2. Run Experiment Locally

We also provide a recipe for local experiments.

Compared with the Google Colab online script, it has following advantages:

  • It can be faster and more stable than online Google Colab (free version) if you have fast GPUs.
  • It is basically the original code we used for our paper, so it should reproduce the exact numbers in the paper.

Step 1. Clone or download this repository and set it as the working directory, create a virtual environment and install the dependencies.

cd vocalsound/ 
python3 -m venv venv-vs
source venv-vs/bin/activate
pip install -r requirements.txt 

Step 2. Download the VocalSound dataset and process it.

cd data/
wget https://www.dropbox.com/s/c5ace70qh1vbyzb/vs_release_16k.zip?dl=0 -O vs_release_16k.zip
unzip vs_release_16k.zip
cd ../src
python prep_data.py

# you can provide a --data_dir augment if you download the data somewhere else
# python prep_data.py --data_dir absolute_path/data

Step 3. Run the baseline experiment

chmod 777 run.sh
./run.sh

# or slurm user
#sbatch run.sh

We test both options before this release, you should get similar accuracies.

Accuracy (%) Colab Script Open In Colab Local Script ICASSP Paper
Validation Set 91.1 90.2 90.1±0.2
All Test Set 91.6 90.6 90.5±0.2
Test Age 18-25 93.4 92.3 91.5±0.3
Test Age 26-48 90.8 90.0 90.1±0.2
Test Age 49-80 92.2 90.2 90.9±1.6
Test Male 89.8 89.6 89.2±0.5
Test Female 93.4 91.6 91.9±0.1
Model Implementation torchvision EfficientNet PSLA EfficientNet PSLA EfficientNet
Batch Size 80 100 100
GPU Google Colab Free 4X Titan 4X Titan
Training Time (30 Epochs) ~2.5 Hours ~1 Hour ~1 Hour

Contact

If you have a question, please bring up an issue (preferred) or send me an email [email protected].

Issues
  • The avg precision is too low, my result is wrong?

    The avg precision is too low, my result is wrong?

    I follow your instruction to run the project with the release vocalsound dataset, and I get the result as follows. The avg precision is too low, and my result is wrong? image

    bug 
    opened by joewale 1
  • FSD50K

    FSD50K

    Hi,

    Congratulations on the Vocalsound dataset. I admire your work very much. Would you please be kind and also share the baseline experiments with the FSD50K + Vocalsound dataset implementation in the repository as well? Best Regards

    enhancement 
    opened by miladbohlouli 3
Owner
Yuan Gong
Postdoc, MIT CSAIL
Yuan Gong
Code for paper 'Audio-Driven Emotional Video Portraits'.

Audio-Driven Emotional Video Portraits [CVPR2021] Xinya Ji, Zhou Hang, Kaisiyuan Wang, Wayne Wu, Chen Change Loy, Xun Cao, Feng Xu [Project] [Paper] G

null 152 Jun 20, 2022
Accompanying code for our paper "Point Cloud Audio Processing"

Point Cloud Audio Processing Krishna Subramani1, Paris Smaragdis1 1UIUC Paper For the necessary libraries/prerequisites, please use conda/anaconda to

Krishna Subramani 16 Jun 25, 2022
Using python to generate a bat script of repetitive lines of code that differ in some way but can sort out a group of audio files according to their common names

Batch Sorting Using python to generate a bat script of repetitive lines of code that differ in some way but can sort out a group of audio files accord

David Mainoo 1 Oct 29, 2021
Code to work with wave files!

Code to work with wave files!

Mohammad Dori 2 Nov 17, 2021
Code for csig audio deepfake detection

FMFCC Audio Deepfake Detection Solution This repo provides an solution for the 多媒体伪造取证大赛. Our solution achieve the 1st in the Audio Deepfake Detection

BokingChen 9 Jun 4, 2022
A small project where I identify notes and key harmonies in a piece of music and use them further to recreate and generate the same piece of music through Python

A small project where I identify notes and key harmonies in a piece of music and use them further to recreate and generate the same piece of music through Python

null 4 Mar 29, 2022
nicfit 374 Jun 16, 2022
Read music meta data and length of MP3, OGG, OPUS, MP4, M4A, FLAC, WMA and Wave files with python 2 or 3

tinytag tinytag is a library for reading music meta data of MP3, OGG, OPUS, MP4, M4A, FLAC, WMA and Wave files with python Install pip install tinytag

Tom Wallroth 536 Jun 29, 2022
Read music meta data and length of MP3, OGG, OPUS, MP4, M4A, FLAC, WMA and Wave files with python 2 or 3

tinytag tinytag is a library for reading music meta data of MP3, OGG, OPUS, MP4, M4A, FLAC, WMA and Wave files with python Install pip install tinytag

Tom Wallroth 435 Feb 17, 2021
Speech recognition module for Python, supporting several engines and APIs, online and offline.

SpeechRecognition Library for performing speech recognition, with support for several engines and APIs, online and offline. Speech recognition engine/

Anthony Zhang 6.3k Jun 20, 2022
Music player and music library manager for Linux, Windows, and macOS

Ex Falso / Quod Libet - A Music Library / Editor / Player Quod Libet is a music management program. It provides several different ways to view your au

Quod Libet 1.1k Jun 19, 2022
C++ library for audio and music analysis, description and synthesis, including Python bindings

Essentia Essentia is an open-source C++ library for audio analysis and audio-based music information retrieval released under the Affero GPL license.

Music Technology Group - Universitat Pompeu Fabra 2.1k Jun 23, 2022
An app made in Python using the PyTube and Tkinter libraries to download videos and MP3 audio.

yt-dl (GUI Edition) An app made in Python using the PyTube and Tkinter libraries to download videos and MP3 audio. How do I download this? Windows: Fi

null 1 Oct 23, 2021
Small Python application that links a Digico console and Reaper, handling automatic marker insertion and tracking.

Digico-Reaper-Link This is a small GUI based helper application designed to help with using Digico's Copy Audio function with a Reaper DAW used for re

Justin Stasiw 6 Jun 5, 2022
Anki vector Music ❤ is the best and only Telegram VC player with playlists, Multi Playback, Channel play and more

Anki Vector Music ?? A bot that can play music on Telegram Group and Channel Voice Chats Available on telegram as @Anki Vector Music Features ?? Thumb

Damantha Jasinghe 10 Jan 9, 2022
Just-Music - Spotify API Driven Music Web app, that allows to listen and control and share songs

Just Music... Just Music Is A Web APP That Allows Users To Play Song Using Spoti

Ayush Mishra 3 May 1, 2022
Audio fingerprinting and recognition in Python

dejavu Audio fingerprinting and recognition algorithm implemented in Python, see the explanation here: How it works Dejavu can memorize audio by liste

Will Drevo 5.8k Jun 26, 2022
Python library for audio and music analysis

librosa A python package for music and audio analysis. Documentation See https://librosa.org/doc/ for a complete reference manual and introductory tut

librosa 5.2k Jun 27, 2022
?️ Open Source Audio Matching and Mastering

Matching + Mastering = ❤️ Matchering 2.0 is a novel Containerized Web Application and Python Library for audio matching and mastering. It follows a si

Sergey Grishakov 630 Jun 15, 2022