raylab
Reinforcement learning algorithms in RLlib and PyTorch.
Installation
pip install raylab
Quickstart
Raylab provides agents and environments to be used with a normal RLlib/Tune setup. You can an agent's name (from the Algorithms section) to raylab info list to list its top-level configurations:
raylab info list SoftAC
learning_starts: 0
Hold this number of timesteps before first training operation.
policy: {}
Sub-configurations for the policy class.
wandb: {}
Configs for integration with Weights & Biases.
Accepts arbitrary keyword arguments to pass to `wandb.init`.
The defaults for `wandb.init` are:
* name: `_name` property of the trainer.
* config: full `config` attribute of the trainer
* config_exclude_keys: `wandb` and `callbacks` configs
* reinit: True
Don't forget to:
* install `wandb` via pip
* login to W&B with the appropriate API key for your
team/project.
* set the `wandb/project` name in the config dict
Check out the Quickstart for more information:
`https://docs.wandb.com/quickstart`
You can add the --rllib flag to get the descriptions for all the options common to RLlib agents (or Trainers)
Launching experiments can be done via the command line using raylab experiment passing a file path with an agent's configuration through the --config flag. The following command uses the cartpole example configuration file to launch an experiment using the vanilla Policy Gradient agent from the RLlib library.
raylab experiment PG --name PG -s training_iteration 10 --config examples/PG/cartpole_defaults.py
You can also launch an experiment from a Python script normally using Ray and Tune. The following shows how you may use Raylab to perform an experiment comparing different types of exploration for the NAF agent.
import ray
from ray import tune
import raylab
def main():
raylab.register_all_agents()
raylab.register_all_environments()
ray.init()
tune.run(
"NAF",
local_dir="data/NAF",
stop={"timesteps_total": 100000},
config={
"env": "CartPoleSwingUp-v0",
"exploration_config": {
"type": tune.grid_search([
"raylab.utils.exploration.GaussianNoise",
"raylab.utils.exploration.ParameterNoise"
])
}
},
num_samples=10,
)
if __name__ == "__main__":
main()
One can then visualize the results using raylab dashboard, passing the local_dir used in the experiment. The dashboard lets you filter and group results in a quick way.
raylab dashboard data/NAF/
You can find the best checkpoint according to a metric (episode_reward_mean by default) using raylab find-best.
raylab find-best data/NAF/
Finally, you can pass a checkpoint to raylab rollout to see the returns collected by the agent and render it if the environment supports a visual render() method. For example, you can use the output of the find-best command to see the best agent in action.
raylab rollout $(raylab find-best data/NAF/) --agent NAF
Algorithms
| Paper | Agent Name |
| Actor Critic using Kronecker-factored Trust Region | ACKTR |
| Trust Region Policy Optimization | TRPO |
| Normalized Advantage Function | NAF |
| Stochastic Value Gradients | SVG(inf)/SVG(1)/SoftSVG |
| Soft Actor-Critic | SoftAC |
| Streamlined Off-Policy (DDPG) | SOP |
| Model-Based Policy Optimization | MBPO |
| Model-based Action-Gradient-Estimator | MAGE |
Command-line interface
For a high-level description of the available utilities, run raylab --help
Usage: raylab [OPTIONS] COMMAND [ARGS]...
RayLab: Reinforcement learning algorithms in RLlib.
Options:
--help Show this message and exit.
Commands:
dashboard Launch the experiment dashboard to monitor training progress.
episodes Launch the episode dashboard to monitor state and action...
experiment Launch a Tune experiment from a config file.
find-best Find the best experiment checkpoint as measured by a metric.
info View information about an agent's config parameters.
rollout Wrap `rllib rollout` with customized options.
test-module Launch dashboard to test generative models from a checkpoint.
Packages
The project is structured as follows
raylab |-- agents # Trainer and Policy classes |-- cli # Command line utilities |-- envs # Gym environment registry and utilities |-- logger # Tune loggers |-- policy # Extensions and customizations of RLlib's policy API | |-- losses # RL loss functions | |-- modules # PyTorch neural network modules for TorchPolicy |-- pytorch # PyTorch extensions |-- utils # miscellaneous utilities
8 Nov 7, 2022
4 Apr 15, 2022
81 Nov 26, 2022
29.6k Jan 8, 2023
4.7k Jan 1, 2023
535 Nov 15, 2022
724 Jan 4, 2023
185 Dec 29, 2022
4.7k Jan 4, 2023
183 Dec 28, 2022
2.1k Jan 4, 2023
4 Dec 31, 2022
195 Dec 7, 2022
21.8k Jan 9, 2023
7.1k Dec 27, 2022
470 Nov 28, 2022
297 Dec 12, 2022
3k Jan 9, 2023
300 Dec 28, 2022