TONet: Tone-Octave Network for Singing Melody Extraction from Polyphonic Music

Overview

TONet

Introduction

The official implementation of "TONet: Tone-Octave Network for Singing Melody Extraction from Polyphonic Music", in ICASSP 2022

We propose TONet, a plug-and-play model that improves both tone and octave perceptions by leveraging a novel input representation and a novel network architecture. Any CFP-input-based Model can be settled in TONet and lead to possible better performance.

TONet Architecture

Main Results on Extraction Performance

Experiments are done to verify the capability of TONet with various baseline backbone models. Our results show that tone-octave fusion with Tone-CFP can significantly improve the singing voice extraction performance across various datasets -- with substantial gains in octave and tone accuracy.

Results

Getting Started

Download Datasets

After downloading the data, use the txt files in the data folder, and process the CFP feature by feature_extraction.py.

Overwrite the Configuration

The config.py contains all configurations you need to change and set.

Train and Evaluation

python main.py train

python main.py test

Produce the Estimation Digram

Uncomment the write prediction in tonet.py

Estimation

Model Checkpoints

We provide the best TO-FTANet checkpoints in this link. More checkpoints will be uploaded.

Citing

@inproceedings{tonet-ke2022,
  author = {Ke Chen, Shuai Yu, Cheng-i Wang, Wei Li, Taylor Berg-Kirkpatrick, Shlomo Dubnov},
  title = {TONet: Tone-Octave Network for Singing Melody Extraction  from Polyphonic Music},
  booktitle = {{ICASSP} 2022}
}
You might also like...
A small project where I identify notes and key harmonies in a piece of music and use them further to recreate and generate the same piece of music through Python
A small project where I identify notes and key harmonies in a piece of music and use them further to recreate and generate the same piece of music through Python

A small project where I identify notes and key harmonies in a piece of music and use them further to recreate and generate the same piece of music through Python

We built this fully functioning Music player in Python. The music player allows you to play/pause and switch to different songs easily.
We built this fully functioning Music player in Python. The music player allows you to play/pause and switch to different songs easily.

We built this fully functioning Music player in Python. The music player allows you to play/pause and switch to different songs easily.

Royal Music You can play music and video at a time in vc
Royal Music You can play music and video at a time in vc

Royals-Music Royal Music You can play music and video at a time in vc Commands SOON String STRING_SESSION Deployment ๐ŸŽ– Credits โ€ข ๐Ÿ‡ธแดแดสแด€โƒ๐Ÿ‡ฏแด‡แด‡แด› โ€ข ๐Ÿ‡ดา“า“ษช

This is an OverPowered Vc Music Player! Will work for you and play music in Voice Chatz
This is an OverPowered Vc Music Player! Will work for you and play music in Voice Chatz

VcPlayer This is an OverPowered Vc Music Player! Will work for you and play music in Voice Chatz Telegram Voice-Chat Bot [PyTGCalls] โ‡ Requirements โ‡œ

This Is Telegram Music UserBot To Play Music Without Being Admin

This Is Telegram Music UserBot To Play Music Without Being Admin

Just-Music - Spotify API Driven Music Web app, that allows to listen and control and share songs

Just Music... Just Music Is A Web APP That Allows Users To Play Song Using Spoti

Xbot-Music - Bot Play Music and Video in Voice Chat Group Telegram
Xbot-Music - Bot Play Music and Video in Voice Chat Group Telegram

XBOT-MUSIC A Telegram Music+video Bot written in Python using Pyrogram and Py-Tg

Noinoi music is smoothly playing music on voice chat of telegram.
Noinoi music is smoothly playing music on voice chat of telegram.

NOINOI MUSIC BOT โœจ Features Music & Video stream support MultiChat support Playlist & Queue support Skip, Pause, Resume, Stop feature Music & Video do

XA Music Player - Telegram Music Bot
XA Music Player - Telegram Music Bot

XA Music Player Requirements ๐Ÿ“ FFmpeg (Latest) NodeJS nodesource.com (NodeJS 17+) Python (3.10+) PyTgCalls (Lastest) MongoDB (3.12.1) 2nd Telegram Ac

Comments
  • Doubt about harmonics

    Doubt about harmonics

    It is mentioned in the paper "Due to the local receptive field of CNN, most existing methods [7, 8, 9, 10] cannot capture harmonic relationships well.", I lack the knowledge in this regard, if stacking more CNNs, is it possible capture harmonic relationships?

    opened by 980202006 1
  • The dataset ref file is missing

    The dataset ref file is missing

    Hello, I want to reproduce your paper, but I can't download the mir-1k dataset with ref file from the official website and I can't get the ref file of medleydb dataset. I want to ask you if there is any way to obtain these missing files? thank you.

    opened by Jayajy 1
  • The problem of multi-GPU training

    The problem of multi-GPU training

    Hi there,

    When I training the model with multi-GPU by setting gpus=2 in pl.Trainer(), it throws an error: TypeError: cannot pickle 'module' object. How can I solve this problem? Thanks!

        ...
        trainer = pl.Trainer(
            deterministic = True,
            gpus = 2, # <---------
            checkpoint_callback = False,
            max_epochs = config.max_epoch,
            auto_lr_find = True,
            sync_batchnorm=True,
            # check_val_every_n_epoch = 1,
            val_check_interval = 0.25,
        )
        ...
    

    python 3.8.3 torch '1.7.1+cu110' Ubuntu 18.04.5 LTS

    Global seed set to 19961206
    Data List: data/test_adc.txt
    Song Size: 12
    100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 12/12 [00:01<00:00,  9.73it/s]
    [W Context.cpp:69] Warning: torch.set_deterministic is in beta, and its design and  functionality may change in the future. (function operator())
    /data4/chengfang/.conda/envs/melody/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:849: UserWarning: You requested multiple GPUs but did not specify a backend, e.g. `Trainer(strategy="dp"|"ddp"|"ddp2")`. Setting `strategy="ddp_spawn"` for you.
      rank_zero_warn(
    /data4/chengfang/.conda/envs/melody/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/callback_connector.py:147: LightningDeprecationWarning: Setting `Trainer(checkpoint_callback=False)` is deprecated in v1.5 and will be removed in v1.7. Please consider using `Trainer(enable_checkpointing=False)`.
      rank_zero_deprecation(
    GPU available: True, used: True
    TPU available: False, using: 0 TPU cores
    IPU available: False, using: 0 IPUs
    LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]
    Traceback (most recent call last):
      File "/data4/chengfang/.conda/envs/melody/lib/python3.8/runpy.py", line 194, in _run_module_as_main
        return _run_code(code, main_globals, None,
      File "/data4/chengfang/.conda/envs/melody/lib/python3.8/runpy.py", line 87, in _run_code
        exec(code, run_globals)
      File "/data4/chengfang/.vscode-server/extensions/ms-python.python-2022.0.1814523869/pythonFiles/lib/python/debugpy/__main__.py", line 45, in <module>
        cli.main()
      File "/data4/chengfang/.vscode-server/extensions/ms-python.python-2022.0.1814523869/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 444, in main
        run()
      File "/data4/chengfang/.vscode-server/extensions/ms-python.python-2022.0.1814523869/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 285, in run_file
        runpy.run_path(target_as_str, run_name=compat.force_str("__main__"))
      File "/data4/chengfang/.conda/envs/melody/lib/python3.8/runpy.py", line 265, in run_path
        return _run_module_code(code, init_globals, run_name,
      File "/data4/chengfang/.conda/envs/melody/lib/python3.8/runpy.py", line 97, in _run_module_code
        _run_code(code, mod_globals, init_globals,
      File "/data4/chengfang/.conda/envs/melody/lib/python3.8/runpy.py", line 87, in _run_code
        exec(code, run_globals)
      File "/data4/chengfang/project/melodyExtraction/TONet/main.py", line 163, in <module>
        train()
      File "/data4/chengfang/project/melodyExtraction/TONet/main.py", line 97, in train
        trainer.fit(model, train_dataloader, test_dataloaders)
      File "/data4/chengfang/.conda/envs/melody/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 740, in fit
        self._call_and_handle_interrupt(
      File "/data4/chengfang/.conda/envs/melody/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 685, in _call_and_handle_interrupt
        return trainer_fn(*args, **kwargs)
      File "/data4/chengfang/.conda/envs/melody/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 777, in _fit_impl
        self._run(model, ckpt_path=ckpt_path)
      File "/data4/chengfang/.conda/envs/melody/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1199, in _run
        self._dispatch()
      File "/data4/chengfang/.conda/envs/melody/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1279, in _dispatch
        self.training_type_plugin.start_training(self)
      File "/data4/chengfang/.conda/envs/melody/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/ddp_spawn.py", line 173, in start_training
        self.spawn(self.new_process, trainer, self.mp_queue, return_result=False)
      File "/data4/chengfang/.conda/envs/melody/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/ddp_spawn.py", line 201, in spawn
        mp.spawn(self._wrapped_function, args=(function, args, kwargs, return_queue), nprocs=self.num_processes)
      File "/data4/chengfang/.conda/envs/melody/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 199, in spawn
        return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
      File "/data4/chengfang/.conda/envs/melody/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 148, in start_processes
        process.start()
      File "/data4/chengfang/.conda/envs/melody/lib/python3.8/multiprocessing/process.py", line 121, in start
        self._popen = self._Popen(self)
      File "/data4/chengfang/.conda/envs/melody/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
        return Popen(process_obj)
      File "/data4/chengfang/.conda/envs/melody/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
        super().__init__(process_obj)
      File "/data4/chengfang/.conda/envs/melody/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
        self._launch(process_obj)
      File "/data4/chengfang/.conda/envs/melody/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
        reduction.dump(process_obj, fp)
      File "/data4/chengfang/.conda/envs/melody/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
        ForkingPickler(file, protocol).dump(obj)
    TypeError: cannot pickle 'module' object
    
    opened by LCF2764 1
  • A question about the octave

    A question about the octave

    Hello, I see octave_class = 8 in the config.py file and use octave_index = (gds // 60 + 2).long()to calculate octave in thetonet.py. I would like to ask the reason for+2. Thanks!!

    opened by Ayyin 0
Owner
Knut(Ke) Chen
ORZ: { godfather: sweetdum, ufo: zgg, dragon sister: lzl, morning king: corner cafรฉ }
Knut(Ke) Chen
Okaeri-Music is a telegram music bot project, allow you to play music on voice chat group telegram.

Okaeri-Music is a telegram bot project that's allow you to play music on telegram voice chat group

Wahyusaputra 1 Dec 22, 2021
Okaeri-Music is a telegram music bot project, allow you to play music on voice chat group telegram.

??๏ธ PROJECT MUSIC,THIS IS MAINTAINED Okaeri-Music is a telegram bot project that's allow you to play music on telegram voice chat group Features ?? Th

Okaeri-Project 2 Dec 23, 2021
NovaMusic is a music sharing robot. Users can get music and music lyrics using inline queries.

A music sharing telegram robot using Redis database and Telebot python library using Redis database.

Hesam Norin 7 Oct 21, 2022
[Singing Log] Let your program learn to sing!

[Singing Log] Let your program learn to sing! You must have thought this was changelog when you saw the English title, but it's not, it's chร nggฤ“log. What it does is allow your program to print logs and sing at the same time!

้ป„ๅท 22 Sep 3, 2022
Official implementation of A cappella: Audio-visual Singing VoiceSeparation, from BMVC21

Y-Net Official implementation of A cappella: Audio-visual Singing VoiceSeparation, British Machine Vision Conference 2021 Project page: ipcv.github.io

Juan F. Montesinos 12 Oct 22, 2022
Official implementation of A cappella: Audio-visual Singing VoiceSeparation, from BMVC21

Y-Net Official implementation of A cappella: Audio-visual Singing VoiceSeparation, British Machine Vision Conference 2021 Project page: ipcv.github.io

Juan F. Montesinos 12 Oct 22, 2022
Music player - endlessly plays your music

Music player First, if you wonder about what is supposed to be a music player or what makes a music player different from a simple media player, read

Albert Zeyer 482 Dec 19, 2022
Music player and music library manager for Linux, Windows, and macOS

Ex Falso / Quod Libet - A Music Library / Editor / Player Quod Libet is a music management program. It provides several different ways to view your au

Quod Libet 1.2k Jan 7, 2023
SU Music Player โ€” The first open-source PyTgCalls based Pyrogram bot to play music in voice chats

SU Music Player โ€” The first open-source PyTgCalls based Pyrogram bot to play music in voice chats Note Neither this, or PyTgCalls are fully

SU Projects 58 Jan 2, 2023
Oliva music bot help to play vc music

OLIVA V2 ?? Requirements ?? FFmpeg NodeJS nodesource.com Python 3.7+ PyTgCalls Commands ?? For all in group /play - reply to youtube url or song file

SOULใ€…Hา‰Aา‰Cา‰Kา‰Eา‰Rา‰ 2 Oct 22, 2021