RLkit
Reinforcement learning framework and algorithms implemented in PyTorch.
Implemented algorithms:
- Semi-supervised Meta Actor Critic
- Skew-Fit
- example script
- paper
- Documentation
- Requires multiworld to be installed
- Reinforcement Learning with Imagined Goals (RIG)
- See this version of this repository.
- paper
- Temporal Difference Models (TDMs)
- Only implemented in v0.1.2 of RLkit. See Legacy Documentation section below.
- paper
- Documentation
- Hindsight Experience Replay (HER)
- (Double) Deep Q-Network (DQN)
- Soft Actor Critic (SAC)
- example script
- original paper and updated version
- TensorFlow implementation from author
- Includes the "min of Q" method, the entropy-constrained implementation, reparameterization trick, and numerical tanh-Normal Jacbian calcuation.
- Twin Delayed Deep Determinstic Policy Gradient (TD3)
- Advantage Weighted Actor Critic (AWAC)
To get started, checkout the example scripts, linked above.
What's New
Version 0.2
04/25/2019
- Use new
multiworldcode that requires explicit environment registration. - Make installation easier by adding
setup.pyand using defaultconf.py.
04/16/2019
- Log how many train steps were called
- Log
env_infoandagent_info.
04/05/2019-04/15/2019
- Add rendering
- Fix SAC bug to account for future entropy (#41, #43)
- Add online algorithm mode (#42)
04/05/2019
The initial release for 0.2 has the following major changes:
- Remove
Serializableclass and use default pickle scheme. - Remove
PyTorchModuleclass and use nativetorch.nn.Moduledirectly. - Switch to batch-style training rather than online training.
- Makes code more amenable to parallelization.
- Implementing the online-version is straightforward.
- Refactor training code to be its own object, rather than being integrated inside of
RLAlgorithm. - Refactor sampling code to be its own object, rather than being integrated inside of
RLAlgorithm. - Implement Skew-Fit: State-Covering Self-Supervised Reinforcement Learning, a method for performing goal-directed exploration to maximize the entropy of visited states.
- Update soft actor-critic to more closely match TensorFlow implementation:
- Rename
TwinSACto justSAC. - Only have Q networks.
- Remove unnecessary policy regualization terms.
- Use numerically stable Jacobian computation.
- Rename
Overall, the refactors are intended to make the code more modular and readable than the previous versions.
Version 0.1
12/04/2018
- Add RIG implementation
12/03/2018
- Add HER implementation
- Add doodad support
10/16/2018
- Upgraded to PyTorch v0.4
- Added Twin Soft Actor Critic Implementation
- Various small refactor (e.g. logger, evaluate code)
Installation
- Install and use the included Ananconda environment
$ conda env create -f environment/[linux-cpu|linux-gpu|mac]-env.yml
$ source activate rlkit
(rlkit) $ python examples/ddpg.py
Choose the appropriate .yml file for your system. These Anaconda environments use MuJoCo 1.5 and gym 0.10.5. You'll need to get your own MuJoCo key if you want to use MuJoCo.
- Add this repo directory to your
PYTHONPATHenvironment variable or simply run:
pip install -e .
- (Optional) Copy
conf.pytoconf_private.pyand edit to override defaults:
cp rlkit/launchers/conf.py rlkit/launchers/conf_private.py
- (Optional) If you plan on running the Skew-Fit experiments or the HER example with the Sawyer environment, then you need to install multiworld.
DISCLAIMER: the mac environment has only been tested without a GPU.
For an even more portable solution, try using the docker image provided in environment/docker. The Anaconda env should be enough, but this docker image addresses some of the rendering issues that may arise when using MuJoCo 1.5 and GPUs. The docker image supports GPU, but it should work without a GPU. To use a GPU with the image, you need to have nvidia-docker installed.
Using a GPU
You can use a GPU by calling
import rlkit.torch.pytorch_util as ptu
ptu.set_gpu_mode(True)
before launching the scripts.
If you are using doodad (see below), simply use the use_gpu flag:
run_experiment(..., use_gpu=True)
Visualizing a policy and seeing results
During training, the results will be saved to a file called under
LOCAL_LOG_DIR/
/
LOCAL_LOG_DIRis the directory set byrlkit.launchers.config.LOCAL_LOG_DIR. Default name is 'output'.setup_logger.exp_prefix.- inside this folder, you should see a file called
params.pkl. To visualize a policy, run
(rlkit) $ python scripts/run_policy.py LOCAL_LOG_DIR/
/
/params.pkl
or
(rlkit) $ python scripts/run_goal_conditioned_policy.py LOCAL_LOG_DIR/
/
/params.pkl
depending on whether or not the policy is goal-conditioned.
If you have rllab installed, you can also visualize the results using rllab's viskit, described at the bottom of this page
tl;dr run
python rllab/viskit/frontend.py LOCAL_LOG_DIR/<exp_prefix>/
to visualize all experiments with a prefix of exp_prefix. To only visualize a single run, you can do
python rllab/viskit/frontend.py LOCAL_LOG_DIR/<exp_prefix>/<folder name>
Alternatively, if you don't want to clone all of rllab, a repository containing only viskit can be found here. You can similarly visualize results with.
python viskit/viskit/frontend.py LOCAL_LOG_DIR/<exp_prefix>/
This viskit repo also has a few extra nice features, like plotting multiple Y-axis values at once, figure-splitting on multiple keys, and being able to filter hyperparametrs out.
Visualizing a goal-conditioned policy
To visualize a goal-conditioned policy, run
(rlkit) $ python scripts/run_goal_conditioned_policy.py
LOCAL_LOG_DIR/
/
/params.pkl
Launching jobs with doodad
The run_experiment function makes it easy to run Python code on Amazon Web Services (AWS) or Google Cloud Platform (GCP) by using this fork of doodad.
It's as easy as:
from rlkit.launchers.launcher_util import run_experiment
def function_to_run(variant):
learning_rate = variant['learning_rate']
...
run_experiment(
function_to_run,
exp_prefix="my-experiment-name",
mode='ec2', # or 'gcp'
variant={'learning_rate': 1e-3},
)
You will need to set up parameters in config.py (see step one of Installation). This requires some knowledge of AWS and/or GCP, which is beyond the scope of this README. To learn more, more about doodad, go to the repository, which is based on this original repository.
Requests for pull-requests
- Implement policy-gradient algorithms.
- Implement model-based algorithms.
Legacy Code (v0.1.2)
For Temporal Difference Models (TDMs) and the original implementation of Reinforcement Learning with Imagined Goals (RIG), run git checkout tags/v0.1.2.
References
The algorithms are based on the following papers
Offline Meta-Reinforcement Learning with Online Self-Supervision Vitchyr H. Pong, Ashvin Nair, Laura Smith, Catherine Huang, Sergey Levine. arXiv preprint, 2021.
Skew-Fit: State-Covering Self-Supervised Reinforcement Learning. Vitchyr H. Pong*, Murtaza Dalal*, Steven Lin*, Ashvin Nair, Shikhar Bahl, Sergey Levine. ICML, 2020.
Visual Reinforcement Learning with Imagined Goals. Ashvin Nair*, Vitchyr Pong*, Murtaza Dalal, Shikhar Bahl, Steven Lin, Sergey Levine. NeurIPS 2018.
Temporal Difference Models: Model-Free Deep RL for Model-Based Control. Vitchyr Pong*, Shixiang Gu*, Murtaza Dalal, Sergey Levine. ICLR 2018.
Hindsight Experience Replay. Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, Wojciech Zaremba. NeurIPS 2017.
Deep Reinforcement Learning with Double Q-learning. Hado van Hasselt, Arthur Guez, David Silver. AAAI 2016.
Human-level control through deep reinforcement learning. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, Demis Hassabis. Nature 2015.
Soft Actor-Critic Algorithms and Applications. Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, Sergey Levine. arXiv preprint, 2018.
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. ICML, 2018.
Addressing Function Approximation Error in Actor-Critic Methods Scott Fujimoto, Herke van Hoof, David Meger. ICML, 2018.
Credits
This repository was initially developed primarily by Vitchyr Pong, until July 2021, at which point it was transferred to the RAIL Berkeley organization and is primarily maintained by Ashvin Nair. Other major collaborators and contributions:
A lot of the coding infrastructure is based on rllab. The serialization and logger code are basically a carbon copy of the rllab versions.
The Dockerfile is based on the OpenAI mujoco-py Dockerfile.
The SMAC code builds off of the PEARL code, which built off of an older RLKit version.
![[Question] Any idea why SAC loss would diverge?](https://avatars.githubusercontent.com/u/1045411?v=4)
4.7k Jan 4, 2023
183 Dec 28, 2022
4 Dec 31, 2022
4.7k Jan 1, 2023
81 Nov 26, 2022
8 Nov 7, 2022
4 Apr 15, 2022
29.6k Jan 8, 2023
535 Nov 15, 2022
724 Jan 4, 2023
185 Dec 29, 2022
220 Jan 7, 2023
51 Dec 27, 2022
348 Dec 24, 2022
42 Dec 9, 2022
704 Dec 27, 2022
2.2k Jan 8, 2023
470 Nov 28, 2022