A deep learning model for style-specific music generation.

Overview

DeepJ: A model for style-specific music generation

https://arxiv.org/abs/1801.00887

Abstract

Recent advances in deep neural networks have enabled algorithms to compose music that is comparable to music composed by humans. However, few algorithms allow the user to generate music with tunable parameters. The ability to tune properties of generated music will yield more practical benefits for aiding artists, filmmakers, and composers in their creative tasks. In this paper, we introduce DeepJ - an end-to-end generative model that is capable of composing music conditioned on a specific mixture of composer styles. Our innovations include methods to learn musical style and music dynamics. We use our model to demonstrate a simple technique for controlling the style of generated music as a proof of concept. Evaluation of our model using human raters shows that we have improved over the Biaxial LSTM approach.

Requirements

  • Python 3.5

Clone Python MIDI (https://github.com/vishnubob/python-midi) cd python-midi then install using python3 setup.py install.

Then, install other dependencies of this project.

pip install -r requirements.txt

The dataset is not provided in this repository. To train a custom model, you will need to include a MIDI dataset in the data/ folder.

Usage

To train a new model, run the following command:

python train.py

To generate music, run the following command:

python generate.py

Use the help command to see CLI arguments:

python generate.py --help
Comments
  • TypeError: __int__ returned non-int (type NoneType)

    TypeError: __int__ returned non-int (type NoneType)

    After installing all requirements (python-midi and requirements.txt) I get this error message when executing generate.py:

    Using TensorFlow backend.
    Traceback (most recent call last):
      File "generate.py", line 153, in <module>
        main()
      File "generate.py", line 142, in main
        models = build_or_load()
      File "/Users/frederikriedel/Developer/DeepJ/util.py", line 15, in build_or_load
        models = build_models()
      File "/Users/frederikriedel/Developer/DeepJ/model.py", line 149, in build_models
        notes_out = naxis(time_out, chosen, style)
      File "/Users/frederikriedel/Developer/DeepJ/model.py", line 111, in f
        dense_layer_cache[l] = Dense(int(x.get_shape()[3]))
    TypeError: __int__ returned non-int (type NoneType)
    

    Do you maybe know what I'm missing here?

    opened by frogg 17
  • write midi file error

    write midi file error

    Writing file out/samples/output_0.mid
    Traceback (most recent call last):
      File "generate.py", line 153, in <module>
        main()
      File "generate.py", line 150, in main
        write_file('output', generate(models, args.bars, styles))
      File "generate.py", line 134, in write_file
        midi.write_midifile(fpath, mf)
      File "/usr/local/python3/lib/python3.5/site-packages/midi/fileio.py", line 169, in write_midifile
        return write_midifile(out,pattern)
      File "/usr/local/python3/lib/python3.5/site-packages/midi/fileio.py", line 171, in write_midifile
        return writer.write(pattern)
      File "/usr/local/python3/lib/python3.5/site-packages/midi/fileio.py", line 105, in write
        self.write_track(track)
      File "/usr/local/python3/lib/python3.5/site-packages/midi/fileio.py", line 122, in write_track
        buf.extend(self.encode_midi_event(event))
      File "/usr/local/python3/lib/python3.5/site-packages/midi/fileio.py", line 161, in encode_midi_event
        ret.extend(event.data)
    ValueError: byte must be in range(0, 256)
    
    opened by dannywu19910524 4
  • is it normal for loss to increased a lot while training?

    is it normal for loss to increased a lot while training?

    Epoch 60/1000
    10954/10954 [==============================] - 659s - loss: 0.0475
    Epoch 61/1000
    10954/10954 [==============================] - 658s - loss: 0.0474
    Epoch 62/1000
    10954/10954 [==============================] - 658s - loss: 0.0473
    Epoch 63/1000
    10954/10954 [==============================] - 659s - loss: 0.0471
    Epoch 64/1000
    10954/10954 [==============================] - 658s - loss: 0.0472
    Epoch 65/1000
    10954/10954 [==============================] - 658s - loss: 0.0612
    Epoch 66/1000
    10954/10954 [==============================] - 659s - loss: 0.0958
    Epoch 67/1000
    10954/10954 [==============================] - 657s - loss: 0.0877
    

    you can see around epoch 66 the loss had increased a lot, does it mean hours of training wasted? sry I am a beginner of this... :(

    opened by DefinitlyEvil 4
  • How to write the command line to generate music of different styles?

    How to write the command line to generate music of different styles?

    the genres and stylesc I set are like this:

    genre = [ 'jazz', 'classical', 'hip_hop' ]

    styles = [ [ 'data/jazz/CharlieParker', 'data/jazz/DavidLiebman', 'data/jazz/JJJohnson' ], [ 'data/classical/beethoven', 'data/classical/holst', 'data/classical/stravinsky' 'data/classical/sullivan' ], [ 'data/hip_hop/50_cent', 'data/hip_hop/ja_rule', 'data/hip_hop/pitbull', 'data/hip_hop/will_smith', ] ]

    How to write the command line to generate music of different styles.

    I use " python generate.py -h" to get the optional arguments,it says: --styles STYLES [STYLES ...] Styles to mix together

    but i can't figure out the specific usage of the style control:

    i have tried " python generate.py --styles classical"and " python generate.py --styles classical[beethoven holst], both showing "error: argument --styles: invalid int value: 'beethoven'"

    opened by JucyCherry 3
  • how is train data fold like

    how is train data fold like

    the folder data/ only contains midi? I encountered error when i train model with data folder only contains midi. So i wonder if there is some misunderstanding about the train data I built. I'd appreciated if U can help me with the problem.(ฅ´ω`ฅ)

    opened by JucyCherry 3
  • Unable to execute with any TF/Keras combination (wt/wo GPU)

    Unable to execute with any TF/Keras combination (wt/wo GPU)

    Hello,

    I've tried running this model with TF 1.6 /1.4 /1.2 (with and without GPU) and Keras 2.0 and am getting the stack trace below [ubuntu 16.04, python3.6, data folder with examples] when i execute the train.py script

    Unable to load model from file. Loading data Training Traceback (most recent call last): File "train.py", line 32, in main() File "train.py", line 16, in main train(models) File "train.py", line 29, in train models[0].fit(train_data, train_labels, epochs=1000, callbacks=cbs, batch_size=BATCH_SIZE) File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 1405, in fit batch_size=batch_size) File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 1295, in _standardize_user_data exception_prefix='model input') File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 121, in _standardize_input_data str(array.shape)) ValueError: Error when checking model input: expected input_1 to have 4 dimensions, but got array with shape (0, 1)

    I tried downgrading Keras to 1.xx but the code uses Keras 2.0 syntax. can you please look into it?

    opened by slave2sync 3
  • build_or_load(): TypeError: __int__ returned non-int (type NoneType)

    build_or_load(): TypeError: __int__ returned non-int (type NoneType)

    build_or_load()

    Traceback (most recent call last): File "util.py", line 36, in build_or_load() File "util.py", line 15, in build_or_load models = build_models() File "/Users/chenqi/workspace/DeepJ/model.py", line 148, in build_models notes_out = naxis(time_out, chosen, style) File "/Users/chenqi/workspace/DeepJ/model.py", line 110, in f dense_layer_cache[l] = Dense(int(x.get_shape()[3])) TypeError: int returned non-int (type NoneType)


    shift_chosen = Lambda(lambda x: tf.pad(x[:, :, :-1, :], [[0, 0], [0, 0], [1, 0], [0, 0]]))(chosen)
    # shift_chosen: shape=(?, 128, 48, 3) 
    shift_chosen = Reshape((time_steps, NUM_NOTES, -1))(shift_chosen)
    # shift_chosen: shape=(?, 128, 48, ?)
    x = Concatenate(axis=3)([x, shift_chosen])
    # x: shape=(?, 128, 48, ?)
    

    then int(x.get_shape()[3]) throw TypeError: int returned non-int (type NoneType)

    opened by chenqi1990 3
  • python-midi in python3?

    python-midi in python3?

    In Ubuntu 16.04 64 with NV tesla P4 box, I can install & run python-midi 0.2.3 in python2,but can not do the same with python3, because " No module named container".

    opened by wuyongyi 2
  • Infinite loading

    Infinite loading

    Website https://deepj.ai/ shows loading sign way too long.

    Console shows errors:

    Failed to load resource: the server responded with a status of 522 ()
    https://server.deepj.ai/stream.mp3?length=1000&seed=0&Baroque=0.021820007785793072&Classical=0.8729091205455646&Romantic=0.5977987382883878&Modern=0.49897429867853504
    
    Failed to load https://server.deepj.ai/stream.mp3?length=1000&seed=0&Baroque=0.021820007785793072&Classical=0.8729091205455646&Romantic=0.5977987382883878&Modern=0.49897429867853504: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'https://deepj.ai' is therefore not allowed access. The response had HTTP status code 522.
    
    opened by DzyubSpirit 1
  • site is forcing https while the api call is to http, causing error and no sound

    site is forcing https while the api call is to http, causing error and no sound

    on the console: GET http://18.222.90.74/stream.mp3?length=1000&seed=0&Baroque=0.0982352937586819&Classical=0.7525334207246455&Romantic=0.8340969462751091&Modern=0.8574476342870909 net::ERR_CONNECTION_REFUSED

    opened by gilamran 1
  • Help needed : Getting error TypeError: __int__ returned non-int (type NoneType) when calling train.py

    Help needed : Getting error TypeError: __int__ returned non-int (type NoneType) when calling train.py

    I am on MAC OS X 10.3.1 Using TensorFlow backend. Traceback (most recent call last): File "train.py", line 32, in main() File "train.py", line 15, in main models = build_or_load() File "/Users/swarnaananthan/storycircles/ai_music/DeepJ/util.py", line 15, in build_or_load models = build_models() File "/Users/swarnaananthan/storycircles/ai_music/DeepJ/model.py", line 149, in build_models notes_out = naxis(time_out, chosen, style) File "/Users/swarnaananthan/storycircles/ai_music/DeepJ/model.py", line 111, in f dense_layer_cache[l] = Dense(int(x.get_shape()[3])) TypeError: int returned non-int (type NoneType)

    Please help me in fixing the issue.

    opened by swarna-a-26 1
  • Unable to load model from file.

    Unable to load model from file.

    Hi, when i use the python generate.py command, it always said 'Unable to load model from file.', but the provided model is there and the file path is right, so why it can't load? Also, although it can't load model, but it can generate one or two midi file and then begins to throw error. The error message is like this:

    Writing file out\samples\output_0.mid Traceback (most recent call last): File "generate.py", line 153, in main() File "generate.py", line 150, in main write_file('output', generate(models, args.bars, styles)) File "generate.py", line 134, in write_file midi.write_midifile(fpath, mf) File "D:\anaconda\anaconda\lib\site-packages\midi-0.2.3-py3.7.egg\midi\fileio.py", line 162, in write_midifile File "D:\anaconda\anaconda\lib\site-packages\midi-0.2. write_file('output', generate(models, args.bars, sty les)) File "generate.py", line 134, in write_file midi.write_midifile(fpath, mf) File "D:\anaconda\anaconda\lib\site-packages\midi-0.2. 3-py3.7.egg\midi\fileio.py", line 162, in write_midifile File "D:\anaconda\anaconda\lib\site-packages\midi-0.2. 3-py3.7.egg\midi\fileio.py", line 110, in write File "D:\anaconda\anaconda\lib\site-packages\midi-0.2. 3-py3.7.egg\midi\fileio.py", line 124, in write_track File "D:\anaconda\anaconda\lib\site-packages\midi-0.2. 3-py3.7.egg\midi\fileio.py", line 152, in encode_midi_ev ent ValueError: byte must be in range(0, 256)

    Other people have the same error, but seems no one knows how to solves this. so my question is: 1.how can i load model correctly? 2.if i haven't load the model, then why it can generate some midi results?How does it being generated? 3.how can i fix this ValueError? Thank you!

    opened by KDcx35 2
  • Summary of the versions to be used

    Summary of the versions to be used

    Can anybody please summarize the versions of all the different libraries that we have to use in this. I have been for the past few hours trying to run this, but always an error comes up. Currently, am stuck at "Failed to load the native TensorFlow runtime".

    opened by shreyasv11 0
  • Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED

    Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED

    after installing requirements, I encountered this error(image below) while running generator.py Screenshot from 2020-01-02 16-58-57

    TensorFlow == 1.15.0

    TensorFlow-gpu == 1.13.1

    Keras == 2.0.0

    ubuntu = 18.04

    NVIDIA-SMI 410.48

    opened by MahdiKoochali 0
  • midi_util.py can't generate replay matrix correctly, the replay matrix is always all zeros.

    midi_util.py can't generate replay matrix correctly, the replay matrix is always all zeros.

    Hi, i'm try to use midi_util.py to extract play, replay, volume matrix from piano-midi.de datasets, but i find that the replay matrix is always the all zeros array. How can I deal with this?

    opened by CSMT201986 0
  • Missing music module in distribution.py

    Missing music module in distribution.py

    Hi i was trying to install the music package using conda but couldn't find it. Is the music file written by you or its a inbuilt python program? I am using the tensorflow version of deepj

    opened by syedrafee 1
Owner
Henry Mao
CTO @ Altum Inc. Entrepreneur, AI Researcher
Henry Mao
PyTorch implementation of MuseMorphose, a Transformer-based model for music style transfer.

MuseMorphose This repository contains the official implementation of the following paper: Shih-Lun Wu, Yi-Hsuan Yang MuseMorphose: Full-Song and Fine-

Yating Music, Taiwan AI Labs 142 Jan 8, 2023
Fast Neural Style for Image Style Transform by Pytorch

FastNeuralStyle by Pytorch Fast Neural Style for Image Style Transform by Pytorch This is famous Fast Neural Style of Paper Perceptual Losses for Real

Bengxy 81 Sep 3, 2022
Emotional conditioned music generation using transformer-based model.

This is the official repository of EMOPIA: A Multi-Modal Pop Piano Dataset For Emotion Recognition and Emotion-based Music Generation. The paper has b

hung anna 96 Nov 9, 2022
PyTorch implementation of ECCV 2020 paper "Foley Music: Learning to Generate Music from Videos "

Foley Music: Learning to Generate Music from Videos This repo holds the code for the framework presented on ECCV 2020. Foley Music: Learning to Genera

Chuang Gan 30 Nov 3, 2022
E2e music remastering system - End-to-end Music Remastering System Using Self-supervised and Adversarial Training

End-to-end Music Remastering System This repository includes source code and pre

Junghyun (Tony) Koo 37 Dec 15, 2022
Style transfer, deep learning, feature transform

FastPhotoStyle License Copyright (C) 2018 NVIDIA Corporation. All rights reserved. Licensed under the CC BY-NC-SA 4.0 license (https://creativecommons

NVIDIA Corporation 10.9k Jan 2, 2023
The personal repository of the work: *DanceNet3D: Music Based Dance Generation with Parametric Motion Transformer*.

DanceNet3D The personal repository of the work: DanceNet3D: Music Based Dance Generation with Parametric Motion Transformer. Dataset and Results Pleas

南嘉Nanga 36 Dec 21, 2022
PyTorch implementation of "Learn to Dance with AIST++: Music Conditioned 3D Dance Generation."

Learn to Dance with AIST++: Music Conditioned 3D Dance Generation. Installation pip install -r requirements.txt Prepare Dataset bash data/scripts/pre

Zj Li 8 Sep 7, 2021
Project for music generation system based on object tracking and CGAN

Project for music generation system based on object tracking and CGAN The project was inspired by MIDINet: A Convolutional Generative Adversarial Netw

null 1 Nov 21, 2021
😇A pyTorch implementation of the DeepMoji model: state-of-the-art deep learning model for analyzing sentiment, emotion, sarcasm etc

------ Update September 2018 ------ It's been a year since TorchMoji and DeepMoji were released. We're trying to understand how it's being used such t

Hugging Face 865 Dec 24, 2022
A multi-functional library for full-stack Deep Learning. Simplifies Model Building, API development, and Model Deployment.

chitra What is chitra? chitra (चित्र) is a multi-functional library for full-stack Deep Learning. It simplifies Model Building, API development, and M

Aniket Maurya 210 Dec 21, 2022
This is the official Pytorch implementation of the paper "Diverse Motion Stylization for Multiple Style Domains via Spatial-Temporal Graph-Based Generative Model"

Diverse Motion Stylization (Official) This is the official Pytorch implementation of this paper. Diverse Motion Stylization for Multiple Style Domains

Soomin Park 28 Dec 16, 2022
MAGMA - a GPT-style multimodal model that can understand any combination of images and language

MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuning Authors repo (alphabetical) Constantin (CoEich), Mayukh (Mayukh

Aleph Alpha GmbH 331 Jan 3, 2023
Variational Attention: Propagating Domain-Specific Knowledge for Multi-Domain Learning in Crowd Counting (ICCV, 2021)

DKPNet ICCV 2021 Variational Attention: Propagating Domain-Specific Knowledge for Multi-Domain Learning in Crowd Counting Baseline of DKPNet is availa

null 19 Oct 14, 2022
Source code for the GPT-2 story generation models in the EMNLP 2020 paper "STORIUM: A Dataset and Evaluation Platform for Human-in-the-Loop Story Generation"

Storium GPT-2 Models This is the official repository for the GPT-2 models described in the EMNLP 2020 paper [STORIUM: A Dataset and Evaluation Platfor

Nader Akoury 27 Dec 20, 2022
A weakly-supervised scene graph generation codebase. The implementation of our CVPR2021 paper ``Linguistic Structures as Weak Supervision for Visual Scene Graph Generation''

README.md shall be finished soon. WSSGG 0 Overview 1 Installation 1.1 Faster-RCNN 1.2 Language Parser 1.3 GloVe Embeddings 2 Settings 2.1 VG-GT-Graph

Keren Ye 35 Nov 20, 2022
Image-generation-baseline - MUGE Text To Image Generation Baseline

MUGE Text To Image Generation Baseline Requirements and Installation More detail

null 23 Oct 17, 2022
Reference implementation of code generation projects from Facebook AI Research. General toolkit to apply machine learning to code, from dataset creation to model training and evaluation. Comes with pretrained models.

This repository is a toolkit to do machine learning for programming languages. It implements tokenization, dataset preprocessing, model training and m

Facebook Research 408 Jan 1, 2023
In this project we investigate the performance of the SetCon model on realistic video footage. Therefore, we implemented the model in PyTorch and tested the model on two example videos.

Contrastive Learning of Object Representations Supervisor: Prof. Dr. Gemma Roig Institutions: Goethe University CVAI - Computational Vision & Artifici

Dirk Neuhäuser 6 Dec 8, 2022