A package for music online and offline rhythmic information analysis including music Beat, downbeat, tempo and meter tracking.

Overview

BeatNet

A package for music online and offline rhythmic information analysis including music Beat, downbeat, tempo and meter tracking.

PyPI CC BY 4.0

PWC PWC PWC PWC PWC PWC

This repository contains the user package and the source code of the Monte Carlo particle flitering inference model of the "BeatNet" music online joint beat/downbeat/tempo/meter tracking system. The arxiv version of the original ISMIR-2021 paper:

arXiv 2108.03576

In addition to the proposed online inference, we added madmom's DBN beat/downbeat inference model for the offline usages. Note that, the offline model still utilize BeatNet's neural network rather than that of Madmom which leads to better performance and significantly faster results.

Note: All models are trained using pytorch and are included in the models folder. In order to recieve the training script and the datasets data/feature handlers, shoot me an email at mheydari [at] ur.rochester.edu

System Input:

Raw audio waveform

System Output:

A vector including beats and downbeats columns, respectively with the following shape: numpy_array(num_beats, 2).

Installation command:

Approach #1: Installing binaries from the pypi website:

pip install BeatNet

Approach #2: Installing directly from the Git repository:

pip install git+https://github.com/mjhydri/BeatNet

Usage example:

From BeatNet.BeatNet import BeatNet

estimator = BeatNet(1) 

Output = estimator.process("music file directory", inference_model= 'PF', plot = True)

A brief video tutorial of the system (Overview):

Easy song


In order to demonstrate the performance of the system for different beat/donbeat tracking difficulties, here are three video demo examples :

1: Song Difficulty: Easy

Easy song

2: Song difficulty: Medium

Easy song

3: Song difficulty: Veteran

Easy song

Acknowledgements:

For the input feature extraction and implementing of the beat state space, Librosa and Madmom libraries are ustilzed. Many thanks for their great jobs. This work has been partially supported by the National Science Foundation grants 1846184 and DGE-1922591.

References:

M. Heydari, F. Cwitkowitz, and Z. Duan, “BeatNet:CRNN and particle filtering for online joint beat down-beat and meter tracking,” in Proc. of the 22th Intl. Conf.on Music Information Retrieval (ISMIR), 2021.

M. Heydari and Z. Duan, “Don’t Look Back: An online beat tracking method using RNN and enhanced particle filtering,” in Proc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP), 2021.

Comments
  • what is the difference of the three feature extraction model?

    what is the difference of the three feature extraction model?

    Hi,Great work! I realize the parameter of the 'model'. image I test the model on the same wav but they get different result. image

    from BeatNet.BeatNet import BeatNet
    import  numpy as np
    def get_bpm(inp):
        begin = inp[0][0]
        durations = []
        for line in inp[1:]:
            durations.append(line[0] - begin)
            begin = line[0]
        return 60 / np.mean(durations)
    
    for i in range(1,4):
        estimator = BeatNet(i, mode='offline', inference_model='PF', thread=False)
    
        Output = estimator.process("bpm_tes1.wav")
        # print(Output)
        print(i, get_bpm(Output))
    
    opened by 980202006 7
  • Restrictive Numba dependency makes Numpy type hints non-descriptive

    Restrictive Numba dependency makes Numpy type hints non-descriptive

    Would it be possible now to update the dependency of Numba to a later version that supports numpy 1.23? Without it, type hints of ndarray is limited to just that, ndarray, when IDEs infer variable types. The later versions allow inferring ndarray[shape, type] such as ndarray[(Any, 3), float] which is very useful for code readability.

    I'm aware of the restriction from librosa that was commented back in 2021, so just wondering if theit misleading (fake) support was fixed?

    Feel free to close this if it's still not possible. Thanks.

    opened by mat100payette 1
  • PyAudio Input overflowed

    PyAudio Input overflowed

    Hello,

    I can run with below modification.


    def activation_extractor_stream(self): # TODO: ''' Streaming window Given the training input window's origin set to center, this streaming data formation causes 0.084 (s) delay compared to the trained model that needs to be fixed. '''
    with torch.no_grad():

                    hop = self.stream.read(self.log_spec_hop_length,exception_on_overflow = False)
    

    opened by AltinayGrass 1
  • Low-quality audio makes detection worse

    Low-quality audio makes detection worse

    Hey, thanks for your sharing. It is a great work.

    I found that when I use 16000Hz audio I get worse results than 22050Hz.(audio from the same music) Inputs are all automatically resampled to 22050 Hz. How can I do better when I only have low quality audio of 16000Hz.

    opened by doreen08 1
  • Invalid device string: 'cuda:cpu' error: bug in model.py

    Invalid device string: 'cuda:cpu' error: bug in model.py

    Hey, thanks for your sharing your work. It is very appreciated!

    After a clean installation I wasn't able to run the example code provided. I tried digging around the code to see if I could fix it for you.

    The error I kept getting was

    RuntimeError: Invalid device string: 'cuda:cpu'

    This code is the problem:

    image

    I changed it to this: image

    and I got it working

    image

    Here is the code.

        def change_device(self, device=None):
            """
            Change the device and load the model onto the new device.
    
            Parameters
            ----------
            device : string or None, optional (default None)
              Device to load model onto
            """
            if device is None:
                # If the function is called without a device, use the current device
                device = self.device
            elif not torch.cuda.is_available():
                device = torch.device('cpu')
            else:
            # Create the appropriate device object
                device = torch.device(f'cuda:{device}')
    
            # Change device field
            self.device = device
            # Load the transcription model onto the device
            self.to(self.device)
    

    I could clean the code a little and open a pull request with this change, if you are open to that. In any case, maybe this issue will help others to make your great project work on their computer.

    Cheers.

    opened by karen-pal 1
  • M1 Mac Support?

    M1 Mac Support?

    I've been wondering if anyone has got this awesome looking package working on M1 - I cannot for the life of me figure out which versions of numba and llvm to use. Brew doesn't let you install a version of llvm compatible with the versions of numba specified.

    I wonder if I should checkout the repo, change some of the version requirements and cross my fingers 😂

    Alternatively is this possible inside docker - will portaudio work inside a docker container?

    opened by aphillipo 4
  • which numpy version should i use?

    which numpy version should i use?

    I used numpy==1.20.3 ,but it reported that ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject,I tried to upgrade numpy, but numba 0.54.1 requires numpy<1.21,>=1.17, but you have numpy 1.23.1 which is incompatible. and reported ImportError: Numba needs NumPy 1.20 or less

    opened by altctrl00 4
  • How to get bpm state space value?

    How to get bpm state space value?

    Running offline mode on audio file produces beat and downbeat array. But I can't access any intermediate state space result. I want to get a scalar bpm value or bpm posterior.

    opened by atodniAr 3
  • add demo script

    add demo script

    This is just a basic demo script that was useful for me while testing the model for different audios and configs.

    Providing a --embed flag will create a blank video clip based on the music that displays the beat & downbeat activations accordingly.

    If you find this useful I can add corresponding docs.

    opened by rlleshi 0
Owner
Mojtaba Heydari
Ph.D. student at Audio Information Retrieval (AIR) Lab-University of Rochester, Research Intern at SiriusXM/Pandora
Mojtaba Heydari
METER: Multimodal End-to-end TransformER

METER Code and pre-trained models will be publicized soon. Citation @article{dou2021meter, title={An Empirical Study of Training End-to-End Vision-a

Zi-Yi Dou 257 Jan 6, 2023
An open source bike computer based on Raspberry Pi Zero (W, WH) with GPS and ANT+. Including offline map and navigation.

Pi Zero Bikecomputer An open-source bike computer based on Raspberry Pi Zero (W, WH) with GPS and ANT+ https://github.com/hishizuka/pizero_bikecompute

hishizuka 264 Jan 2, 2023
Python package for multiple object tracking research with focus on laboratory animals tracking.

motutils is a Python package for multiple object tracking research with focus on laboratory animals tracking. Features loads: MOTChallenge CSV, sleap

Matěj Šmíd 2 Sep 5, 2022
The source code for the Cutoff data augmentation approach proposed in this paper: "A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation".

Cutoff: A Simple Data Augmentation Approach for Natural Language This repository contains source code necessary to reproduce the results presented in

Dinghan Shen 49 Dec 22, 2022
Technical experimentations to beat the stock market using deep learning :chart_with_upwards_trend:

DeepStock Technical experimentations to beat the stock market using deep learning. Experimentations Deep Learning Stock Prediction with Daily News Hea

Keon 449 Dec 29, 2022
This is the codebase for Diffusion Models Beat GANS on Image Synthesis.

This is the codebase for Diffusion Models Beat GANS on Image Synthesis.

OpenAI 3k Dec 26, 2022
Codebase for Diffusion Models Beat GANS on Image Synthesis.

Codebase for Diffusion Models Beat GANS on Image Synthesis.

Katherine Crowson 128 Dec 2, 2022
An integration of several popular automatic augmentation methods, including OHL (Online Hyper-Parameter Learning for Auto-Augmentation Strategy) and AWS (Improving Auto Augment via Augmentation Wise Weight Sharing) by Sensetime Research.

An integration of several popular automatic augmentation methods, including OHL (Online Hyper-Parameter Learning for Auto-Augmentation Strategy) and AWS (Improving Auto Augment via Augmentation Wise Weight Sharing) by Sensetime Research.

null 45 Dec 8, 2022
Omnidirectional Scene Text Detection with Sequential-free Box Discretization (IJCAI 2019). Including competition model, online demo, etc.

Box_Discretization_Network This repository is built on the pytorch [maskrcnn_benchmark]. The method is the foundation of our ReCTs-competition method

Yuliang Liu 266 Nov 24, 2022
Generalized Decision Transformer for Offline Hindsight Information Matching

Generalized Decision Transformer for Offline Hindsight Information Matching [arxiv] If you use this codebase for your research, please cite the paper:

Hiroki Furuta 35 Dec 12, 2022
Much faster than SORT(Simple Online and Realtime Tracking), a little worse than SORT

QSORT QSORT(Quick + Simple Online and Realtime Tracking) is a simple online and realtime tracking algorithm for 2D multiple object tracking in video s

Yonghye Kwon 8 Jul 27, 2022
The official repo for OC-SORT: Observation-Centric SORT on video Multi-Object Tracking. OC-SORT is simple, online and robust to occlusion/non-linear motion.

OC-SORT Observation-Centric SORT (OC-SORT) is a pure motion-model-based multi-object tracker. It aims to improve tracking robustness in crowded scenes

Jinkun Cao 325 Jan 5, 2023
Implementation of the CVPR 2021 paper "Online Multiple Object Tracking with Cross-Task Synergy"

Online Multiple Object Tracking with Cross-Task Synergy This repository is the implementation of the CVPR 2021 paper "Online Multiple Object Tracking

null 54 Oct 15, 2022
Joint detection and tracking model named DEFT, or ``Detection Embeddings for Tracking.

DEFT: Detection Embeddings for Tracking DEFT: Detection Embeddings for Tracking, Mohamed Chaabane, Peter Zhang, J. Ross Beveridge, Stephen O'Hara

Mohamed Chaabane 253 Dec 18, 2022
Tracking code for the winner of track 1 in the MMP-Tracking Challenge at ICCV 2021 Workshop.

Tracking Code for the winner of track1 in MMP-Trakcing challenge This repository contains our tracking code for the Multi-camera Multiple People Track

DamoCV 29 Nov 13, 2022
Tracking Pipeline helps you to solve the tracking problem more easily

Tracking_Pipeline Tracking_Pipeline helps you to solve the tracking problem more easily I integrate detection algorithms like: Yolov5, Yolov4, YoloX,

VNOpenAI 32 Dec 21, 2022
Quadruped-command-tracking-controller - Quadruped command tracking controller (flat terrain)

Quadruped command tracking controller (flat terrain) Prepare Install RAISIM link

Yunho Kim 4 Oct 20, 2022
High performance, easy-to-use, and scalable machine learning (ML) package, including linear model (LR), factorization machines (FM), and field-aware factorization machines (FFM) for Python and CLI interface.

What is xLearn? xLearn is a high performance, easy-to-use, and scalable machine learning package that contains linear model (LR), factorization machin

Chao Ma 3k Jan 3, 2023