Benchmark

The benchmark results of IQL and NFQ have been added to d3rlpy-benchmarks. Plus, the results of the more random seeds up to 10 have been added to all algorithms. The benchmark results are more reliable now.

Documentation

• More descriptions have been added to Finetuning tutorial page.
• Offline Policy Selection tutorial page has been added

Enhancements

• cloudpickle and GPUUtil dependencies have been removed.
• gaussian likelihood computation for MOPO becomes more mathematically right (thanks @tominku )

MDPDataset

The timestep alignment is now exactly the same as D4RL:

# observations = [o_1, o_2, ..., o_n]
observations = np.random.random((1000, 10))

# actions = [a_1, a_2, ..., a_n]
actions = np.random.random((1000, 10))

# rewards = [r(o_1, a_1), r(o_2, a_2), ...]
rewards = np.random.random(1000)

# terminals = [t(o_1, a_1), t(o_2, a_2), ...]
terminals = ...

where r(o, a) is the reward function and t(o, a) is the terminal function.

The reason of this change is that the many users were confused with the difference between d3rlpy and D4RL. But, now it's aligned in the same way. This change might break your dataset.

Algorithms

• Neural Fitted Q-iteration (NFQ)
  • https://link.springer.com/chapter/10.1007/11564096_32

Enhancements

• AWAC, CRR and IQL use a non-squashed gaussian policy function.
• The more tutorial pages have been added to the documentation.
• The software design page has been added to the documentation.
• The reproduction script for IQL has been added.
• The progress bar in online training is visually improved in Jupyter Notebook #161 (thanks, @aiueola )
• The nan checks have been added to MDPDataset.
• The target_reduction_type and bootstrap options have been removed.

Bugfix

• The unnecessary test conditions have been removed
• Typo in dataset.pyx has been fixed #167 (thanks, @zbzhu99 )
• The details of IQL implementation have been fixed.

It's proud to announce that v1.0.0 has been finally released! The first version was released in Aug 2020 under the support of the IPA MITOU program. At the first release, d3rlpy only supported a few algorithms and did not even support online training. After months of constructive feedbacks and insights from the users and the community, d3rlpy has been established as the first offline deep RL library with many online and offline algorithms support and unique features. The next chapter also starts towards the ambitious v2.0.0 today. Please stay tuned for the next announcement!

NeurIPS 2021 Offline RL Workshop

The workshop paper about d3rlpy has been presented at the NeurIPS 2021 Offline RL Workshop. URL: https://arxiv.org/abs/2111.03788

Benchmarks

The full benchmark results are finally available at d3rlpy-benchmarks.

Algorithms

• Implicit Q-Learning (IQL)
  • https://arxiv.org/abs/2110.06169

Enhancements

• deterministic option is added to collect method
• rollout_return metrics is added to online training
• random_steps is added to fit_online method
• --save option is added to d3rlpy CLI commands (thanks, @pstansell )
• multiplier option is added to reward normalizers
• many reproduction scripts are added
• policy_type option is added to BC
• get_atari_transition function is added for the Atari 2600 offline benchmark procedure

Bugfix

• document fix (thanks, @araffin )
• Fix TD3+BC's actor loss function
• Fix gaussian noise for TD3 exploration

Roadmap towards v2.0.0

• Sophisticated config system using dataclasses
• Dump configuration and model parameters in a single file
• Change MDPDataset format to align with D4RL datasets
• Support large dataset
• Support tuple observation
• Support large-scale data-parallel offline training
• Support large-scale distributed online training
• Support Transformer architecture (e.g. Decision Transformer)
• Speed up training with torch.jit.script and CUDA Graphs
• Change library name to represent the unification of offline and online

Algorithm

• TD3+BC
  • https://arxiv.org/abs/2106.06860

RewardScaler

From this version, the preprocessors are available for the rewards, which allow you to normalize, standardize and clip the reward values.

import d3rlpy

# normalize
cql = d3rlpy.algos.CQL(reward_scaler="min_max")

# standardize
cql = d3rlpy.algos.CQL(reward_scaler="standardize")

# clip (you can't use string alias)
cql = d3rlpy.algos.CQL(reward_scaler=d3rlpy.preprocessing.ClipRewardScaler(-1.0, 1.0))

copy_policy_from and copy_q_function_from methods

In the scenario of finetuning, you might want to initialize SAC's policy function with the pretrained CQL's policy function to boost the initial performance. From this version, you can do that as follows:

import d3rlpy

# pretrain with static dataset
cql = d3rlpy.algos.CQL()
cql.fit(...)

# transfer the policy function
sac = d3rlpy.algos.SAC()
sac.copy_policy_from(cql)

# you can also transfer the Q-function
sac.copy_q_function_from(cql)

# finetuning with online algorithm
sac.fit_online(...)

Enhancements

• show messages for skipping model builds
• add alpha parameter option to DiscreteCQL
• keep counting the number of gradient steps
• allow expanding MDPDataset with the larger discrete actions (thanks, @jamartinh )
• callback function is called every gradient step (previously, it's called every epoch)

Bugfix

• FQE's loss function has been fixed (thanks for the report, @guyk1971)
• fix documentation build (thanks, @astrojuanlu)
• fix d4rl dataset conversion for MDPDataset (this will have a significant impact on the performance for d4rl dataset)

Algorithm

• Conservative Offline Model-Based Optimization (COMBO)
  • https://arxiv.org/abs/2102.08363

Drop data augmentation feature

From this version, the data augmentation feature has been dropped. The reason for this is that the feature introduces a lot of code complexity. In order to make d3rlpy support many algorithms and keep it as simple as possible, the feature was dropped. Instead, TorchMiniBatch was internally introduced, and all algorithms become more simple.

collect method

In offline RL experiments, data collection plays an important role especially when you try new tasks. From this version, collect method is finally available.

import d3rlpy
import gym

# prepare environment
env = gym.make('Pendulum-v0')

# prepare algorithm
sac = d3rlpy.algos.SAC()

# prepare replay buffer
buffer = d3rlpy.online.buffers.ReplayBuffer(maxlen=100000, env=env)

# start data collection without updates
sac.collect(env, buffer)

# export to MDPDataset
dataset = buffer.to_mdp_dataset()

# save as file
dataset.dump('pendulum.h5')

Along with this change, random policies are also introduced. These are useful to collect dataset with random policy.

# continuous action-space
policy = d3rlpy.algos.RandomPolicy()

# discrete action-space
policy = d3rlpy.algos.DiscreteRandomPolicy()

Enhancements

• CQL and BEAR become closer to the official implementations
• callback argument has been added to algorithms
• random dataset has been added to cartpole and pendulum dataset
  • you can specify it via dataset_type='random' at get_cartpole and get_pendulum method

Bugfix

• fix action normalization at predict_value method (thanks, @navidmdn )
• fix seed settings at reproduction codes

What's missing before v1.00?

Currently, I'm benchmarking all algorithms with d4rl dataset. Through the experiments, I realized that it's very difficult to reproduce the table reported in the paper because they actually didn't reveal full hyper-parameters, which are tuned to each dataset. So I gave up reproducing the table, and start producing numbers with the official codes to see if d3rlpy's result matches.

Algorithms

New algorithms are introduced in this version.

• Critic Regularized Regression (CRR)
  • https://arxiv.org/abs/2006.15134
• Model-based Offline Policy Optimization (MOPO)
  • https://arxiv.org/abs/2005.13239

Model-based RL

Previously, model-based RL has been supported. The model-based specific logic was implemented in dynamics side. This approach enabled us to combine model-based algorithms with arbitrary model-free algorithms. However, this requires complex designs to implement the recent model-based RL. So, the dynamics interface was refactored and the MOPO is the first algorithm to show how d3rlpy supports model-based RL algorithms.

# train dynamics model
from d3rlpy.datasets import get_pendulum
from d3rlpy.dynamics import ProbabilisticEnsembleDynamics
from d3rlpy.metrics.scorer import dynamics_observation_prediction_error_scorer
from d3rlpy.metrics.scorer import dynamics_reward_prediction_error_scorer
from d3rlpy.metrics.scorer import dynamics_prediction_variance_scorer
from sklearn.model_selection import train_test_split

dataset, _ = get_pendulum()

train_episodes, test_episodes = train_test_split(dataset)

dynamics = d3rlpy.dynamics.ProbabilisticEnsembleDynamics(learning_rate=1e-4, use_gpu=True)

dynamics.fit(train_episodes,
             eval_episodes=test_episodes,
             n_epochs=100,
             scorers={
                'observation_error': dynamics_observation_prediction_error_scorer,
                'reward_error': dynamics_reward_prediction_error_scorer,
                'variance': dynamics_prediction_variance_scorer,
             })

# train Model-based RL algorithm
from d3rlpy.algos import MOPO

# give mopo as generator argument.
mopo = MOPO(dynamics=dynamics)

mopo.fit(dataset, n_steps=100000)

enhancements

• fitter method has been implemented (thanks @jamartinh )
• tensorboard_dir repleces tensorboard flag at fit method (thanks @navidmdn )
• show warning messages when the unused arguments are passed
• show comprehensive error messages when action-space is not compatible
• fit method accepts MDPDataset object
• dropout option has been implemented in encoders
• add appropriate __repr__ methods to show pretty outputs when print(algo)
• metrics collection is refactored

bugfix

• fix core dumped errors by fixing numpy version
• fix CQL backup

Command Line Interface

New commands are added in this version.

record

You can record the video of the evaluation episodes without coding anything.

$ d3rlpy record d3rlpy_logs/CQL_20201224224314/model_100.pt --env-id HopperBulletEnv-v0

# record wrapped environment
$ d3rlpy record d3rlpy_logs/Discrete_CQL_20201224224314/model_100.pt \
    --env-header 'import gym; env = d3rlpy.envs.Atari(gym.make("BreakoutNoFrameskip-v4"), is_eval=True)'

play

You can run the evaluation episodes with rendering images.

# record simple environment
$ d3rlpy play d3rlpy_logs/CQL_20201224224314/model_100.pt --env-id HopperBulletEnv-v0

# record wrapped environment
$ d3rlpy play d3rlpy_logs/Discrete_CQL_20201224224314/model_100.pt \
    --env-header 'import gym; env = d3rlpy.envs.Atari(gym.make("BreakoutNoFrameskip-v4"), is_eval=True)'

data-point mask for bootstrapping

Ensemble training for Q-functions has been shown as a powerful method to achieve robust training. Previously, bootstrap option has been available for algorithms. But, the mask for Q-function loss is randomly created every time when the batch is sampled.

In this version, create_mask option is available for MDPDataset and ReplayBuffer, which will create a unique mask at each data-point.

# offline training
dataset = d3rlpy.dataset.MDPDataset(observations, actions, rewards, terminals, create_mask=True, mask_size=5)
cql = d3rlpy.algos.CQL(n_critics=5, bootstrap=True, target_reduction_type='none')
cql.fit(dataset)

# online training
buffer = d3rlpy.online.buffers.ReplayBuffer(1000000, create_mask=True, mask_size=5)
sac = d3rlpy.algos.SAC(n_critics=5, bootstrap=True, target_reduction_type='none')
sac.fit_online(env, buffer)

As you noticed above, target_reduction_type is newly introduced to specify how to aggregate target Q values. In the standard Soft Actor-Critic, the target_reduction_type='min'. If you choose none, each ensemble Q-function uses its own target value, which is similar to what Bootstrapped DQN does.

better module access

From this version, you can navigate to all modules through d3rlpy.

# previously
from d3rlpy.datasets import get_cartpole
dataset = get_cartpole()

# v0.70
import d3rlpy
dataset = d3rlpy.datasets.get_cartpole()

new logger style

From this version, structlog is internally used to print information instead of raw print function. This allows us to emit more structural information. Furthermore, you can control what to show and what to save to the file if you overwrite logger configuration.

enhancements

• soft_q_backup option is added to CQL.
• Paper Reproduction page has been added to the documentation in order to show the performance with the paper configuration.
• commit method at D3RLPyLogger returns metrics (thanks, @jamartinh )

bugfix

• fix epoch count in offline training.
• fix total_step count in online training.
• fix typos at documentation (thanks, @pstansell )

CLI

record command is newly introduced in this version. You can record videos of evaluation episodes with the saved model.

$ d3rlpy record d3rlpy_logs/CQL_20210131144357/model_100.pt --env-id Hopper-v2

You can also use the wrapped environment.

$ d3rlpy record d3rlpy_logs/DQN_online_20210130170041/model_1000.pt \
  --env-header 'import gym; from d3rlpy.envs import Atari; env = Atari(gym.make("BreakoutNoFrameskip-v4"), is_eval=True)'

bugfix

• fix saving models every step in fit_online method
• fix Atari wrapper to reproduce the paper result
• fix CQL and BEAR algorithms

logo

New logo images are made for d3rlpy 🎉

| standard | inverted | |:-:|:-:| |||

ActionScaler

ActionScaler provides action scaling pre/post-processing for continuous control algorithms. Previously actions must be in between [-1.0, 1.0]. From now on, you don't need to care about the range of actions.

from d3rlpy.cql import CQL

cql = CQL(action_scaler='min_max')  # just pass action_scaler argument

handling timeout episodes

Episodes terminated by timeouts should not be clipped at bootstrapping. From this version, you can specify episode boundaries as well as the terminal flags.

from d3rlpy.dataset import MDPDataset

observations = ...
actions = ...
rewards = ...
terminals = ... # this indicates the environmental termination
episode_terminals = ... # this indicates episode boundaries

datasets = MDPDataset(observations, actions, rewards, terminals, episode_terminals)

# if episode_terminals are omitted, terminals will be used to specify episode boundaries
# datasets = MDPDataset(observations, actions, rewards, terminals) 

In online training, you can specify this option via timelimit_aware flag.

from d3rlpy.sac import SAC

env = gym.make('Hopper-v2') # make sure if the environment is wrapped by gym.wrappers.Timelimit

sac = SAC()
sac.fit_online(env, timelimit_aware=True) # this flag is True by default

reference: https://arxiv.org/abs/1712.00378

batch online training

When training with computationally expensive environments such as robotics simulators or rich 3D games, it will take a long time to finish due to the slow environment steps. To solve this, d3rlpy supports batch online training.

from d3rlpy.algos import SAC
from d3rlpy.envs import AsyncBatchEnv

if __name__ == '__main__':  # this is necessary if you use AsyncBatchEnv
    env = AsyncBatchEnv([lambda: gym.make('Hopper-v2') for _ in range(10)])  # distributing 10 environments in different processes

    sac = SAC(use_gpu=True)
    sac.fit_batch_online(env) # train with 10 environments concurrently

docker image

Pre-built d3rlpy docker image is available in DockerHub.

$ docker run -it --gpus all --name d3rlpy takuseno/d3rlpy:latest bash

enhancements

• BEAR algorithm is updated based on the official implementation
  • new mmd_kernel option is available
• to_mdp_dataset method is added to ReplayBuffer
• ConstantEpsilonGreedy explorer is added
• d3rlpy.envs.ChannelFirst wrapper is added (thanks for reporting, @feyza-droid )
• new dataset utility function d3rlpy.datasets.get_d4rl is added
  • this is handling timeouts inside the function
• offline RL paper reproduction codes are added
• smoothed moving average plot at d3rlpy plot CLI function (thanks, @pstansell )
• user-friendly messages for assertion errors
• better memory consumption
• save_interval argument is added to fit_online

bugfix

• core dumps are fixed in Google Colaboratory tutorials
• typos in some documentations (thanks for reporting, @pstansell )

minor fix

• add typing-extensions depdency
• update MANIFEST.in

typing

Now, d3rlpy is fully type-annotated not only for the better use of this library but also for the better contribution experiences.

• mypy and pylint check the type consistency and code quality.
• due to a lot of changes to add type annotations, there might be degradation that is not detected by linters.

CLI

v0.50 introduces the new command-line interface, d3rlpy command that helps you to do more without any efforts. For now, d3rlpy provides the following commands.

# plot CSV data
$ d3rlpy plot d3rlpy_logs/XXX/YYY.csv

# plot CSV data
$ d3rlpy plot-all d3rlpy_logs/XXX

# export the save model as inference formats (e.g. ONNX, TorchScript)
$ d3rlpy export d3rlpy_logs/XXX/model_YYY.pt

enhancements

• faster CPU to GPU transfer
  • this change makes online training x2 faster
• make IQN Q function more precise based on the paper

documentation

• Add doc about SB3 integration ( thanks, @araffin )

Algorithm

• Policy in Latent Action Space (PLAS)
  • https://arxiv.org/abs/2011.07213

Off-Policy Evaluation

Off-policy evaluation (OPE) is a method to evaluate policy performance only with the offline dataset.

# train policy
from d3rlpy.algos import CQL
from d3rlpy.datasets import get_pybullet
dataset, env = get_pybullet('hopper-bullet-mixed-v0')
cql = CQL()
cql.fit(dataset.episodes)

# Off-Policy Evaluation
from d3rlpy.ope import FQE
from d3rlpy.metrics.scorer import soft_opc_scorer
from d3rlpy.metrics.scorer import initial_state_value_estimation_scorer
fqe = FQE(algo=cql)
fqe.fit(dataset.episodes,
           eval_episodes=dataset.episodes
           scorers={
               'soft_opc': soft_opc_scorer(1000),
               'init_value': initial_state_value_estimation_scorer
           })

• Fitted Q-Evaluation
  • https://arxiv.org/abs/2007.09055

Q Function Factory

d3rlpy provides flexible controls over Q functions through Q function factory. Following this change, the previous q_func_type argument was renamed to q_func_factory.

from d3rlpy.algos import DQN
from d3rlpy.q_functions import QRQFunctionFactory

# initialize Q function factory
q_func_factory = QRQFunctionFactory(n_quantiles=32)

# give it to algorithm object
dqn = DQN(q_func_factory=q_func_factory)

You can pass Q function name as string too.

dqn = DQN(q_func_factory='qr')

You can also make your own Q function factory. Currently, these are the supported Q function factory.

• MeanQFunctionFactory
• QRQFunctionFactory
• IQNQFunctionFactory
• FQFQFunctionFactory

EncoderFactory

• DenseNet architecture (only for vector observation)
  • https://arxiv.org/abs/2010.09163

from d3rlpy.algos import DQN

dqn = DQN(encoder_factory='dense')

N-step TD calculation

d3rlpy supports N-step TD calculation for ALL algorithms. You can pass n_steps arugment to configure this parameters.

from d3rlpy.algos import DQN

dqn = DQN(n_steps=5) # n_steps=1 by default

Paper reproduction scripts

d3rlpy supports many algorithms including online and offline paradigms. Originally, d3rlpy is designed for industrial practitioners. But, academic research is still important to push deep reinforcement learning forward. Currently, there are online DQN-variant reproduction codes.

• DQN
• Double DQN
• QR-DQN
• IQN
• FQF

The evaluation results will be also available soon.

enhancements

• build_with_dataset and build_with_env methods are added to algorithm objects
• shuffle flag is added to fit method (thanks, @jamartinh )

Algorithms

• Support the discrete version of Soft Actor-Critic
  • https://arxiv.org/abs/1910.07207
• fit_online has n_steps argument instead of n_epochs for the complete reproduction of the papers.

OptimizerFactory

d3rlpy provides more flexible controls for optimizer configuration via OptimizerFactory.

from d3rlpy.optimizers import AdamFactory
from d3rlpy.algos import DQN

dqn = DQN(optim_factory=AdamFactory(weight_decay=1e-4))

See more at https://d3rlpy.readthedocs.io/en/v0.40/references/optimizers.html .

EncoderFactory

d3rlpy provides more flexible controls for the neural network architecture via EncoderFactory.

from d3rlpy.algos import DQN
from d3rlpy.encoders import VectorEncoderFactory

# encoder factory
encoder_factory = VectorEncoderFactory(hidden_units=[300, 400], activation='tanh')

# set OptimizerFactory
dqn = DQN(encoder_factory=encoder_factory)

Also you can build your own encoders.

import torch
import torch.nn as nn

from d3rlpy.encoders import EncoderFactory

# your own neural network
class CustomEncoder(nn.Module):
    def __init__(self, obsevation_shape, feature_size):
        self.feature_size = feature_size
        self.fc1 = nn.Linear(observation_shape[0], 64)
        self.fc2 = nn.Linear(64, feature_size)

    def forward(self, x):
        h = torch.relu(self.fc1(x))
        h = torch.relu(self.fc2(h))
        return h

    # THIS IS IMPORTANT!
    def get_feature_size(self):
        return self.feature_size

# your own encoder factory
class CustomEncoderFactory(EncoderFactory):
    TYPE = 'custom' # this is necessary

    def __init__(self, feature_size):
        self.feature_size = feature_size

    def create(self, observation_shape, action_size=None, discrete_action=False):
        return CustomEncoder(observation_shape, self.feature_size)

    def get_params(self, deep=False):
        return {
            'feature_size': self.feature_size
        }

dqn = DQN(encoder_factory=CustomEncoderFactory(feature_size=64))

See more at https://d3rlpy.readthedocs.io/en/v0.40/references/network_architectures.html .

Stable Baselines 3 wrapper

• Now d3rlpy is partially compatible with Stable Baselines 3.
  • https://github.com/takuseno/d3rlpy/blob/master/d3rlpy/wrappers/sb3.py
• More documentations will be available soon.

bugfix

• fix the memory leak problem at fit_online.
  • Now, you can train online algorithms with the big replay buffer size for the image observation.
• fix preprocessing at CQL.
• fix ColorJitter augmentation.

installation

PyPi

• From this version, d3rlpy officially supports Windows.
• The binary packages for each platform are built in GitHub Actions. And they are uploaded, which means that you don't have to install Cython to install this package from PyPi.

Anaconda

• From previous version, d3rlpy is available in conda-forge.

This version introduces hotfix.

• ⚠️ Fix the significant bug in the case of online training with image observation.

This version introduces minor changes.

• Move n_epochs arguments to fit method.
• Fix scikit-learn compatibility issues.
• Fix zero-division error during online training.

Algorithm

• Support Advantage-Weighted Actor-Critic (AWAC)
  • https://arxiv.org/abs/2006.09359
• fit_online method is available as a convenient alias to d3rlpy.online.iterators.train function.
• unnormalizing action problem is fixed at AWR.

Metrics

• The following metrics are available.
  • initial_state_value_estimation_scorer
    • https://arxiv.org/abs/1906.01624
  • soft_opc_scorer
    • https://arxiv.org/abs/2007.09055

⚠️ MDPDataset

• d3rlpy.dataset module is now implemented with Cython in order to speed up memory copies.
• Following operations are significantly faster than the previous version.
  • creating TransitionMiniBatch object
  • frame stacking via n_frames argument
  • lambda return calculation at AWR algorithms
• This change approximately makes Atari training 6% faster.

Algorithm

• Support Advantage-Weighted Regression (AWR)
  • https://arxiv.org/abs/1910.00177
• n_frames option is added to all algorithms
  • n_frames option controls frame stacking for image observation
• eval_results_ property is added to all algorithms
  • evaluation results can be retrieved from eval_results_ after training.

MDPDataset

• prev_transition and next_transition properties are added to d3rlpy.dataset.Transition.
  • these properties are used for frame stacking and Monte-Carlo returns calculation at AWR.

Document

• new tutorial page is added

Support ONNX export

Now, the trained policy can be exported as ONNX as well as TorchScript

cql.save_policy('policy.onnx', as_onnx=True)

Support more data augmentations

• data augmentations for vector obsrevation
• ColorJitter augmentation for image observation

• support model-based algorithm
  • Model-based Offline Policy Optimization
• support data augmentation (for image observation)
  • Data-reguralized Q-learning
• a lot of improvements
  • more dataset statistics
  • more options to customize neural network architecture
  • optimize default learning rates
  • etc

• online algorithms
  • Deep Q-Network (DQN)
  • Double DQN
  • Deep Deterministic Policy Gradients (DDPG)
  • Twin Delayed Deep Deterministic Policy Gradients (TD3)
  • Soft Actor-Critic (SAC)
• data-driven algorithms
  • Batch-Constrained Q-leearning (BCQ)
  • Bootstrapping Error Accumulation Reduction (BEAR)
  • Conservative Q-Learning (CQL)
• Q functions
  • mean
  • Quantile Regression
  • Implicit Quantile Network
  • Fully-parametrized Quantile Function (experimental)