An OpenAI-Gym Package for Training and Testing Reinforcement Learning algorithms with OpenSim Models

Overview

Logo

Authors: Utkarsh A. Mishra and Dr. Dimitar Stanev

Advisors: Dr. Dimitar Stanev and Prof. Auke Ijspeert, Biorobotics Laboratory (BioRob), EPFL

Video Playlist: https://www.youtube.com/playlist?list=PLDvnH871wUkFPOcCKcsTN6ZzzjNZOVlt_

The bioimiitation-gym package is a python package that provides a gym environment for training and testing OpenSim models. The gym environment is based on the OpenAI gym package.

This work is towards a framework aimed towards learning to imitate human gaits. Humans exhibit movements like walking, running, and jumping in the most efficient manner, which served as the source of motivation for this project. Skeletal and Musculoskeletal human models were considered for motions in the sagittal and frontal plane, and results from both were compared exhaustively. While skeletal models are driven with motor actuation, musculoskeletal models perform through muscle-tendon actuation.

Baseline Architecture

Model-free reinforcement learning algorithms were used to optimize inverse dynamics control actions to satisfy the objective of imitating a reference motion along with secondary objectives of minimizing effort in terms of power spent by motors and metabolic energy consumed by the muscles. On the one hand, the control actions for the motor actuated model is the target joint angles converted into joint torques through a Proportional-Differential controller. While on the other hand, the control actions for the muscle-tendon actuated model is the muscle excitations converted implicitly to muscle activations and then to muscle forces which apply moments on joints. Muscle-tendon actuated models were found to have superiority over motor actuation as they are inherently smooth due to muscle activation dynamics and don't need any external regularizers.

Results

All the results and analysis are presented in an illustrative, qualitative, and quantitative manner.

Installation

Please follow the instructions in the installation.md file to install the package.

Environment in the bioimitation-gym package

All environments in the bioimitation-gym package are provided in the biomitation_envs/imitation_envs/envs directory. They are majorly divided into two categories:

  • muscle environments: These are the environments that are used for training the muscle tendon unit actuated model.
  • torque environments: These are the environments that are used for training the torque actuate model.

Further, 2D / planar and 3D / spatial environments are provided for each category. The tasks covered in each of the sub-categories are as follows:

  • Walking
  • Running
  • Jumping
  • Prosthetic Walking with a locked knee joint for the left leg
  • Walking with a typical Cerebel Palsy defect

The following 2D muscle actuated environment names can be used based on the package:

  • MuscleWalkingImitation2D-v0
  • MuscleRunningImitation2D-v0
  • MuscleJumpingImitation2D-v0
  • MuscleLockedKneeImitation2D-v0

The following 3D muscle actuated environment names can be used based on the package:

  • MuscleWalkingImitation3D-v0
  • MuscleRunningImitation3D-v0
  • MuscleJumpingImitation3D-v0
  • MuscleLockedKneeImitation3D-v0
  • MusclePalsyImitation3D-v0

The following 2D torque actuated environment names can be used based on the package:

  • TorqueWalkingImitation2D-v0
  • TorqueRunningImitation2D-v0
  • TorqueJumpingImitation2D-v0
  • TorqueLockedKneeImitation2D-v0

The following 3D torque actuated environment names can be used based on the package:

  • TorqueWalkingImitation3D-v0
  • TorqueRunningImitation3D-v0
  • TorqueJumpingImitation3D-v0
  • TorqueLockedKneeImitation3D-v0

Usage Instructions

The complete bioimitation directory consists of the following sub-directories:

  • imitation_envs: This directory contains the data and environments associated with the package.
  • learning_algorithm: This directory contains the learning algorithm used for several experiments. The code is the modified version of original SAC algorithm and is taken from the open source implementation of ikostrikov/jaxrl.

More information on the subdirectories can be found in their respective README files (if any).

The package is mostly based on the highly scalable and distributed reinforcement learning framework Ray RLLIB. The template scipts to train and test the models are provided in the tests directory.

To run a RLLIB training script, run the following command:

python tests/sample_rllib_training.py  --env_name MuscleWalkingImitation2D-v0

You can change the algorithm configurations in the configs directory. The configs/train_default.py file contains the default configuration for the train script and the configs/test_default.py file contains the default configuration for the test script which is:

python tests/sample_rllib_testing.py

The default environment configuration is provided in the configs/env_default.py file. Feel free to change the default configuration as per your needs. A typical script to test the environment is provided in the biomitation_envs/imitation_envs/envs directory is:

import os
from absl import app, flags
from ml_collections import config_flags
import gym
import bioimitation

FLAGS = flags.FLAGS

flags.DEFINE_string('env_name', 'MuscleWalkingImitation2D-v0', 'Name of the environment.')

config_flags.DEFINE_config_file(
    'config',
    'configs/env_default.py',
    'File path to the environment configuration.',
    lock_config=False)

def main(_):

    example_config = dict(FLAGS.config)

    env = gym.make(FLAGS.env_name, config=example_config)

    env.reset()

    for i in range(1000):
        _, _, done, _ = env.step(env.action_space.sample())
        if done:
            env.reset()

if __name__ == '__main__':
    app.run(main)

Don't forget to import the bioimitation package before running the script.

Citation

If you use this work in your research, please cite the following as:

@misc{
    mishra2021bioimitation,
    title = {BioImitation-Gym: A OpenAI-Gym Package for Training and Testing Reinforcement Learning algorithms with OpenSim Models},
    author = {Utkarsh A. Mishra and Dimitar Stanev and Auke Ijspeert},
    year = {2021},
    url = {https://github.com/UtkarshMishra/bioimitation-gym}
}
@article{mishra2021learning,
  title={Learning Control Policies for Imitating Human Gaits},
  author={Utkarsh A. Mishra},
  journal={arXiv preprint arXiv:2106.15273},
  year={2021}
}

References

[1] OsimRL project: https://osim-rl.kidzinski.com/

[2] OpenSim: https://github.com/opensim-org/opensim-core and https://opensim.stanford.edu/

[3] OpenAI Gym: https://gym.openai.com/

[4] Ray RLLIB: https://ray.readthedocs.io/en/latest/

[6] ikostrikov/jaxrl: https://github.com/ikostrikov/jaxrl

You might also like...
Scripts of Machine Learning Algorithms from Scratch. Implementations of machine learning models and algorithms using nothing but NumPy with a focus on accessibility. Aims to cover everything from basic to advance.
Scripts of Machine Learning Algorithms from Scratch. Implementations of machine learning models and algorithms using nothing but NumPy with a focus on accessibility. Aims to cover everything from basic to advance.

Algo-ScriptML Python implementations of some of the fundamental Machine Learning models and algorithms from scratch. The goal of this project is not t

This is the official repo for TransFill:  Reference-guided Image Inpainting by Merging Multiple Color and Spatial Transformations at CVPR'21. According to some product reasons, we are not planning to release the training/testing codes and models. However, we will release the dataset and the scripts to prepare the dataset. Conservative Q Learning for Offline Reinforcement Reinforcement Learning in JAX
Conservative Q Learning for Offline Reinforcement Reinforcement Learning in JAX

CQL-JAX This repository implements Conservative Q Learning for Offline Reinforcement Reinforcement Learning in JAX (FLAX). Implementation is built on

Reinforcement-learning - Repository of the class assignment questions for the course on reinforcement learning

DSE 314/614: Reinforcement Learning This repository containing reinforcement lea

AutoPentest-DRL: Automated Penetration Testing Using Deep Reinforcement Learning
AutoPentest-DRL: Automated Penetration Testing Using Deep Reinforcement Learning

AutoPentest-DRL: Automated Penetration Testing Using Deep Reinforcement Learning AutoPentest-DRL is an automated penetration testing framework based o

A PyTorch Lightning solution to training OpenAI's CLIP from scratch.
A PyTorch Lightning solution to training OpenAI's CLIP from scratch.

train-CLIP šŸ“Ž A PyTorch Lightning solution to training CLIP from scratch. Goal āš½ Our aim is to create an easy to use Lightning implementation of OpenA

A toolkit for developing and comparing reinforcement learning algorithms.

Status: Maintenance (expect bug fixes and minor updates) OpenAI Gym OpenAI Gym is a toolkit for developing and comparing reinforcement learning algori

PyTorch implementations of deep reinforcement learning algorithms and environments
PyTorch implementations of deep reinforcement learning algorithms and environments

Deep Reinforcement Learning Algorithms with PyTorch This repository contains PyTorch implementations of deep reinforcement learning algorithms and env

Pytorch implementations of popular off-policy multi-agent reinforcement learning algorithms, including QMix, VDN, MADDPG, and MATD3.

Off-Policy Multi-Agent Reinforcement Learning (MARL) Algorithms This repository contains implementations of various off-policy multi-agent reinforceme

Comments
  • The example has no reference data for execution

    The example has no reference data for execution

    This repository looks very useful, but when I installed it to run the example, I couldn't try the package because the following .sto file was not available. This is very unfortunate.

    Can you kindly provide the link to the .sto file below?

    https://github.com/UtkarshMishra04/bioimitation-gym/blob/658304b119c59123e977f37b9801eed6614cdf2d/bioimitation/imitation_envs/envs/muscle/planar/muscle_jumping_imitation_env2D.py#L46-L53

    https://github.com/UtkarshMishra04/bioimitation-gym/blob/658304b119c59123e977f37b9801eed6614cdf2d/bioimitation/imitation_envs/envs/muscle/planar/muscle_locked_knee_imitation_env2D.py#L46-L53

    opened by sk413025 0
Owner
Utkarsh Mishra
Graduate from @iitroorkee (Batch of 2021), programming enthusiast. Reinforcement Learning, Robotics & Self-Driving interests me.
Utkarsh Mishra
Trading Gym is an open source project for the development of reinforcement learning algorithms in the context of trading.

Trading Gym Trading Gym is an open-source project for the development of reinforcement learning algorithms in the context of trading. It is currently

Dimitry Foures 535 Nov 15, 2022
Deep Q Learning with OpenAI Gym and Pokemon Showdown

pokemon-deep-learning An openAI gym project for pokemon involving deep q learning. Made by myself, Sam Little, and Layton Webber. This code captures g

null 2 Dec 22, 2021
Customizable RecSys Simulator for OpenAI Gym

gym-recsys: Customizable RecSys Simulator for OpenAI Gym Installation | How to use | Examples | Citation This package describes an OpenAI Gym interfac

Xingdong Zuo 14 Dec 8, 2022
Manipulation OpenAI Gym environments to simulate robots at the STARS lab

Manipulator Learning This repository contains a set of manipulation environments that are compatible with OpenAI Gym and simulated in pybullet. In par

STARS Laboratory 5 Dec 8, 2022
An OpenAI Gym environment for Super Mario Bros

gym-super-mario-bros An OpenAI Gym environment for Super Mario Bros. & Super Mario Bros. 2 (Lost Levels) on The Nintendo Entertainment System (NES) us

Andrew Stelmach 1 Jan 5, 2022
Reinforcement Learning with Q-Learning Algorithm on gym's frozen lake environment implemented in python

Reinforcement Learning with Q Learning Algorithm Q learning algorithm is trained on the gym's frozen lake environment. Libraries Used gym Numpy tqdm P

null 1 Nov 10, 2021
šŸ„A PyTorch implementation of OpenAI's finetuned transformer language model with a script to import the weights pre-trained by OpenAI

PyTorch implementation of OpenAI's Finetuned Transformer Language Model This is a PyTorch implementation of the TensorFlow code provided with OpenAI's

Hugging Face 1.4k Jan 5, 2023
gym-anm is a framework for designing reinforcement learning (RL) environments that model Active Network Management (ANM) tasks in electricity distribution networks.

gym-anm is a framework for designing reinforcement learning (RL) environments that model Active Network Management (ANM) tasks in electricity distribution networks. It is built on top of the OpenAI Gym toolkit.

Robin Henry 99 Dec 12, 2022
Multi-objective gym environments for reinforcement learning.

MO-Gym: Multi-Objective Reinforcement Learning Environments Gym environments for multi-objective reinforcement learning (MORL). The environments follo

Lucas Alegre 74 Jan 3, 2023
BasicRL: easy and fundamental codes for deep reinforcement learning怂It is an improvement on rainbow-is-all-you-need and OpenAI Spinning Up.

BasicRL: easy and fundamental codes for deep reinforcement learning BasicRL is an improvement on rainbow-is-all-you-need and OpenAI Spinning Up. It is

RayYoh 12 Apr 28, 2022