Starter kit for getting started in the Music Demixing Challenge.

AIcrowd

Last update: Dec 20, 2022

Overview

Music Demixing Challenge - Starter Kit

👉 Challenge page

This repository is the Music Demixing Challenge Submission template and Starter kit!

Clone the repository to compete now!

This repository contains:

Documentation on how to submit your models to the leaderboard
The procedure for best practices and information on how we evaluate your agent, etc.
Starter code for you to get started!

Competition Procedure
How to access and use dataset
How to start participating
How do I specify my software runtime / dependencies?
What should my code structure be like ?
How to make submission
Other concepts
Important links

Competition Procedure

The Music Demixing (MDX) Challenge is an opportunity for researchers and machine learning enthusiasts to test their skills by creating a system able to perform audio source separation.

In this challenge, you will train your models locally and then upload them to AIcrowd (via git) to be evaluated.

The following is a high level description of how this process works

Sign up to join the competition on the AIcrowd website.
Clone this repo and start developing your solution.
Train your models for audio seperation and write prediction code in test.py.
Submit your trained models to AIcrowd Gitlab for evaluation (full instructions below). The automated evaluation setup will evaluate the submissions against the test dataset to compute and report the metrics on the leaderboard of the competition.

How to access and use the dataset

You are allowed to train your system either exclusively on the training part of MUSDB18-HQ dataset or you can use your choice of data. According to the dataset used, you will be eligible for different leaderboards.

👉 Download MUSDB18-HQ dataset

In case you are using external dataset, please mention it in your aicrowd.json.

{
  [...],
  "external_dataset_used": true
}

The MUSDB18 dataset contains 150 songs (100 songs in train and 50 songs in test) together with their seperations in the following manner:

|
├── train
│   ├── A Classic Education - NightOwl
│   │   ├── bass.wav
│   │   ├── drums.wav
│   │   ├── mixture.wav
│   │   ├── other.wav
│   │   └── vocals.wav
│   └── ANiMAL - Clinic A
│       ├── bass.wav
│       ├── drums.wav
│       ├── mixture.wav
│       ├── other.wav
│       └── vocals.wav
[...]

Here the mixture.wav file is the original music on which you need to do audio source seperation.
While bass.wav, drums.wav, other.wav and vocals.wav contain files for your training purposes.
Please note again: To be eligible for Leaderboard A, you are only allowed to train on the songs in train.

How to start participating

Setup

Add your SSH key to AIcrowd GitLab

You can add your SSH Keys to your GitLab account by going to your profile settings here. If you do not have SSH Keys, you will first need to generate one.

Clone the repository

git clone [email protected]:AIcrowd/music-demixing-challenge-starter-kit.git

Install competition specific dependencies!

cd music-demixing-challenge-starter-kit
pip3 install -r requirements.txt

Try out random prediction codebase present in test.py.

How do I specify my software runtime / dependencies ?

We accept submissions with custom runtime, so you don't need to worry about which libraries or framework to pick from.

The configuration files typically include requirements.txt (pypi packages), environment.yml (conda environment), apt.txt (apt packages) or even your own Dockerfile.

You can check detailed information about the same in the 👉 RUNTIME.md file.

What should my code structure be like ?

Please follow the example structure as it is in the starter kit for the code structure. The different files and directories have following meaning:

.
├── aicrowd.json           # Submission meta information - like your username
├── apt.txt                # Packages to be installed inside docker image
├── data                   # Your local dataset copy - you don't need to upload it (read DATASET.md)
├── requirements.txt       # Python packages to be installed
├── test.py                # IMPORTANT: Your testing/prediction code, must be derived from MusicDemixingPredictor (example in test.py)
└── utility                # The utility scripts to provide smoother experience to you.
    ├── docker_build.sh
    ├── docker_run.sh
    ├── environ.sh
    └── verify_or_download_data.sh

Finally, you must specify an AIcrowd submission JSON in aicrowd.json to be scored!

The aicrowd.json of each submission should contain the following content:

{
  "challenge_id": "evaluations-api-music-demixing",
  "authors": ["your-aicrowd-username"],
  "description": "(optional) description about your awesome agent",
  "external_dataset_used": false
}

This JSON is used to map your submission to the challenge - so please remember to use the correct challenge_id as specified above.

How to make submission

👉 SUBMISSION.md

Best of Luck 🎉 🎉

Other Concepts

Time constraints

You need to make sure that your model can do audio seperation for each song within 4 minutes, otherwise the submission will be marked as failed.

Local Run

👉 LOCAL_RUN.md

Contributing

🙏 You can share your solutions or any other baselines by contributing directly to this repository by opening merge request.

Add your implemntation as test_<approach-name>.py
Test it out using python test_<approach-name>.py
Add any documentation for your approach at top of your file.
Import it in predict.py
Create merge request! 🎉 🎉 🎉

Contributors

📎 Important links

💪 Challenge Page: https://www.aicrowd.com/challenges/music-demixing-challenge-ismir-2021

🗣️ Discussion Forum: https://www.aicrowd.com/challenges/music-demixing-challenge-ismir-2021/discussion

🏆 Leaderboard: https://www.aicrowd.com/challenges/music-demixing-challenge-ismir-2021/leaderboards

Comments

README submission template for mediocre submissions

Hello!

Not to be too down on myself - I'm happy with my work and progress - but my place in the leaderboard is pretty low (lower than 30) - i.e. it may not be too interesting of a model to be published, etc. Do I necessarily need to fill the special submission README?

https://github.com/AIcrowd/music-demixing-challenge-starter-kit/blob/master/README_SUBMISSION_TEMPLATE.md

Is it only for the top 10 submissions that will be included in the final writeup/summary?

opened by sevagh 2

Struggling to get setup

Hi,

I'm having trouble making a test submission using the repo.

I am working in a fresh nvcr.io/nvidia/cuda:11.3.0-cudnn8-runtime-centos8 docker image. Following that I have:

Installed GCC, git, nano via Yum.
Installed miniconda.
Generated SSH key and added it to my profile.
Git cloned the repo & installed requirements.
Changed the aicrowd.json to my username.
Run the below:

# Add AIcrowd git remote endpoint
git remote add aicrowd [email protected]:errorfixrepeat/music-demixing-challenge-starter-kit.git
git push aicrowd master

This gives me the error:

Enumerating objects: 186, done.
Counting objects: 100% (186/186), done.
Delta compression using up to 24 threads
Compressing objects: 100% (87/87), done.
Writing objects: 100% (186/186), 5.09 MiB | 1.79 MiB/s, done.
Total 186 (delta 90), reused 186 (delta 90)
remote: Resolving deltas: 100% (90/90), done.
remote: GitLab: LFS objects are missing. Ensure LFS is properly set up or try a manual "git lfs push --all".
To gitlab.aicrowd.com:errorfixrepeat/music-demixing-challenge-starter-kit.git
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to '[email protected]:errorfixrepeat/music-demixing-challenge-starter-kit.git'

opened by errorfixrepeat 2

Error cloning repo

Hi

When I try to clone the repo, I get an error when it gets to the data/test/Mu - Too Bright/bass.wav file

fatal error: unexpected signal during runtime execution [signal 0xb code=0x1 addr=0xb01dfacedebac1e pc=0xca0b0]

I have already seen the posts that suggest trying git lfs pull, which I have tried without joy. See below:

fatal error: still in list

runtime stack: runtime.throw(0x495d50, 0xd) /usr/local/go/src/runtime/panic.go:527 +0x90 runtime.mHeap_AllocSpanLocked(0x6b0ea0, 0x80, 0xc820110000) /usr/local/go/src/runtime/mheap.go:590 +0x40c runtime.mHeap_Alloc_m(0x6b0ea0, 0x80, 0x700100000000, 0xc81fff77ff) /usr/local/go/src/runtime/mheap.go:425 +0x1ac runtime.mHeap_Alloc.func1() /usr/local/go/src/runtime/mheap.go:484 +0x41 runtime.systemstack(0x700008dc6e88) /usr/local/go/src/runtime/asm_amd64.s:278 +0xab runtime.mHeap_Alloc(0x6b0ea0, 0x80, 0x10100000000, 0xc82001d220) /usr/local/go/src/runtime/mheap.go:485 +0x63 runtime.largeAlloc(0x100000, 0xc800000001, 0x1) /usr/local/go/src/runtime/malloc.go:745 +0xb3 runtime.mallocgc.func3() /usr/local/go/src/runtime/malloc.go:634 +0x33 runtime.systemstack(0xc820021500) /usr/local/go/src/runtime/asm_amd64.s:262 +0x79 runtime.mstart() /usr/local/go/src/runtime/proc1.go:674

Git version is 2.31.1 on macOS 10.15.7.

Any help would be much appreciated.

Cheers

Matt

opened by easablade 2
Demucs baseline

This adds the Demucs baseline for the MDX challenge.

I added a page on the demucs repo with specific instructions for MDX: https://github.com/facebookresearch/demucs/blob/master/docs/mdx.md

By the way, it seems the 64 channels Demucs will timeout on the challenge (not sure if that was the error, but one submission failed at 88%). The 48 channels one ran fine.

Thanks a lot for organising this :)

opened by adefossez 1
Wrong index when assigning "sdr_bass" value
https://github.com/AIcrowd/music-demixing-challenge-starter-kit/blob/7802a7d5de37cddccf842e29603b47b03b2f4e47/evaluator/music_demixing.py#L104

Should be replaced by

scores[music_name]["sdr_bass"] = song_score[0]
opened by yoyololicon 1
Better error handling
we assume that the sample rates of the signals and the length of the references and estimates are the same. However often, users might not be aware of this and it would make sense to warn them already in the local test submission.

we have:

https://github.com/AIcrowd/music-demixing-challenge-starter-kit/blob/4cd3be898a9f9ef09692ad82075c5fd38f3ffee9/evaluator/music_demixing.py#L95-L96

I think we should add

a check of the sample rates (the second argument _) these should the same (int(reference_rate) == int(estimate_rate))

a check of the shape of the files (reference.shape == estimate.shape)

enhancement
opened by faroit 0

The format of the wav file is unknown.

I get a RuntimeError when I run the fourth Try out random prediction codebase present in test.py. in Setup. I found that the format of the wav file in data/test was unknown. I opened this wav file in audacity, but it showed up as a file of unknown format. Is this wav file the correct one? Sorry if this is my mistake. The full text of the error is below.

> python3 test.py
Mixture file is present at following location: /home/shun/aicrowd/music-demixing-challenge-starter-kit/data/test/Mu - Too Bright/mixture.wav
Traceback (most recent call last):
  File "/home/shun/aicrowd/music-demixing-challenge-starter-kit/evaluator/music_demixing.py", line 142, in run
    self.evaluation()
  File "/home/shun/aicrowd/music-demixing-challenge-starter-kit/evaluator/music_demixing.py", line 129, in evaluation
    self.prediction(mixture_file_path=self.get_music_file_location(music_name),
  File "test.py", line 52, in prediction
    x, rate = sf.read(mixture_file_path)  # mixture is stereo with sample rate of 44.1kHz
  File "/home/shun/.local/share/virtualenvs/music-demixing-challenge-starter-kit-dMCFg2FA/lib/python3.8/site-packages/soundfile.py", line 256, in read
    with SoundFile(file, 'r', samplerate, channels,
  File "/home/shun/.local/share/virtualenvs/music-demixing-challenge-starter-kit-dMCFg2FA/lib/python3.8/site-packages/soundfile.py", line 629, in __init__
    self._file = self._open(file, mode_int, closefd)
  File "/home/shun/.local/share/virtualenvs/music-demixing-challenge-starter-kit-dMCFg2FA/lib/python3.8/site-packages/soundfile.py", line 1183, in _open
    _error_check(_snd.sf_error(file_ptr),
  File "/home/shun/.local/share/virtualenvs/music-demixing-challenge-starter-kit-dMCFg2FA/lib/python3.8/site-packages/soundfile.py", line 1357, in _error_check
    raise RuntimeError(prefix + _ffi.string(err_str).decode('utf-8', 'replace'))
RuntimeError: Error opening '/home/shun/aicrowd/music-demixing-challenge-starter-kit/data/test/Mu - Too Bright/mixture.wav': File contains data in an unknown format.

Traceback (most recent call last):
  File "test.py", line 64, in <module>
    submission.run()
  File "/home/shun/aicrowd/music-demixing-challenge-starter-kit/evaluator/music_demixing.py", line 148, in run
    raise e
  File "/home/shun/aicrowd/music-demixing-challenge-starter-kit/evaluator/music_demixing.py", line 142, in run
    self.evaluation()
  File "/home/shun/aicrowd/music-demixing-challenge-starter-kit/evaluator/music_demixing.py", line 129, in evaluation
    self.prediction(mixture_file_path=self.get_music_file_location(music_name),
  File "test.py", line 52, in prediction
    x, rate = sf.read(mixture_file_path)  # mixture is stereo with sample rate of 44.1kHz
  File "/home/shun/.local/share/virtualenvs/music-demixing-challenge-starter-kit-dMCFg2FA/lib/python3.8/site-packages/soundfile.py", line 256, in read
    with SoundFile(file, 'r', samplerate, channels,
  File "/home/shun/.local/share/virtualenvs/music-demixing-challenge-starter-kit-dMCFg2FA/lib/python3.8/site-packages/soundfile.py", line 629, in __init__
    self._file = self._open(file, mode_int, closefd)
  File "/home/shun/.local/share/virtualenvs/music-demixing-challenge-starter-kit-dMCFg2FA/lib/python3.8/site-packages/soundfile.py", line 1183, in _open
    _error_check(_snd.sf_error(file_ptr),
  File "/home/shun/.local/share/virtualenvs/music-demixing-challenge-starter-kit-dMCFg2FA/lib/python3.8/site-packages/soundfile.py", line 1357, in _error_check
    raise RuntimeError(prefix + _ffi.string(err_str).decode('utf-8', 'replace'))
RuntimeError: Error opening '/home/shun/aicrowd/music-demixing-challenge-starter-kit/data/test/Mu - Too Bright/mixture.wav': File contains data in an unknown format.

opened by tkeshun 5

Owner

AIcrowd

GitHub

Development Kit for the SoccerNet Challenge

SoccerNetv2-DevKit Welcome to the SoccerNet-V2 Development Kit for the SoccerNet Benchmark and Challenge. This kit is meant as a help to get started w

117 Dec 30, 2022

ManiSkill-Learn is a framework for training agents on SAPIEN Open-Source Manipulation Skill Challenge (ManiSkill Challenge), a large-scale learning-from-demonstrations benchmark for object manipulation.

ManiSkill-Learn ManiSkill-Learn is a framework for training agents on SAPIEN Open-Source Manipulation Skill Challenge, a large-scale learning-from-dem

48 Dec 30, 2022

Music Source Separation; Train & Eval & Inference piplines and pretrained models we used for 2021 ISMIR MDX Challenge.

Music Source Separation with Channel-wise Subband Phase Aware ResUnet (CWS-PResUNet) Introduction This repo contains the pretrained Music Source Separ

100 Dec 25, 2022

Scripts and a shader to get you started on setting up an exported Koikatsu character in Blender.

KK Blender Shader Pack A plugin and a shader to get you started with setting up an exported Koikatsu character in Blender. The plugin is a Blender add