This PR introduces recurrent SAC-discrete algorithm for POMDPs with discrete action space. The code is heavily based on the SAC-discrete open-sourced code https://github.com/ku2482/sac-discrete.pytorch/blob/master/sacd/agent/sacd.py and the SAC-discrete paper https://arxiv.org/abs/1910.07207

We provide two sanity checks on classic gym discrete control environments: CartPole-v0 and LunarLander-v2. The commands for running Markovian and recurrent SAC-discrete algorithms are:

# CartPole
python3 policies/main.py --cfg configs/pomdp/cartpole/f/mlp.yml --target_entropy 0.7 --cuda -1
# CartPole-V
python3 policies/main.py --cfg configs/pomdp/cartpole/v/rnn.yml --target_entropy 0.7 --cuda 0
# Lunalander
python3 policies/main.py --cfg configs/pomdp/lunalander/f/mlp.yml --target_entropy 0.7 --cuda -1
# Lunalander-V
python3 policies/main.py --cfg configs/pomdp/lunalander/v/rnn.yml --target_entropy 0.5 --cuda 0

where target_entropy sets the ratio of target entropy: ratio * log(|A|).

• CartPole: Markovian SAC-discrete is quite sensitive to target_entropy but can solve the task with max return 200:

• CartPole-V: recurrent SAC-discrete is robust to target_entropy and can solve the task with max return 200 within 10 episodes

• Lunalander: Markovian SAC-discrete is sensitive to target_entropy but can solve the task with return over 200:

• Lunalander-V: recurrent SAC-discrete is very sensitive to target_entropy but can nearly solve the task in one target_entropy value:

Hello,

I wonder do you have any thoughts on how to handle that case? Currently, you assume it never happens and the code will raise an error for such cases. However, in some domains like the MiniGrid Lava Crossing, for instance, that might happen quite often. Thanks!

For SACD, can you explain why you do this https://github.com/twni2016/pomdp-baselines/blob/main/policies/models/policy_rnn.py#L235 instead of this (which I think is correct)? https://github.com/twni2016/pomdp-baselines/blob/main/policies/models/policy_rnn.py#L230

This PR is not intended to be merged, but as a showcase for support pixel observation with discrete action space, e.g. Atari games.

We take delayed-catch environment as a sanity check, introduced by IMPALA+SR https://arxiv.org/abs/2102.12425 The environment has only terminal reward and requires long-term memory. It has image size of 1x7x7, discrete action of 3, horizon of ~runs*7. We use a simple image encoder for image observation to replace the MLP encoder for vector observation.

We try delayed-catch with 5, 10, 20, 40 runs. The more the runs, the harder the problems. Below are the learning curves of 10, 20, 40 runs for IMPALA and IMPALA+SR (their Fig. 7b).

Our running command:

# We sweep over the following range
python3 policies/main.py --cfg configs/pomdp/catch/rnn.yml --noautomatic_entropy_tuning --entropy_alpha [0.1,0.01,0.001]

where we found fixed temperature works much better than auto-tuning it with target entropy in this task. (Still a bit strange why this can work but that cannot; auto-tuning will finally has zero actor gradient).

• Delayed-cach with 5 runs: solve it with 100k samples

• Delayed-cach with 10 runs: solve it with 400k samples (vs 50M for IMPALA+SR)

• Delayed-cach with 20 runs: solve it with 700k samples (vs 100M for IMPALA+SR)

• Delayed-cach with 40 runs: after hparam tuning, can solve it with 2M samples (vs 200M for IMPALA+SR)

Different fixed alpha value:

With alpha=0.1:

This PR is a major code refactor on policy and RL part. The idea is to remove RL algorithm specific details from policy*.py, e.g. TD3, SAC, SAC-D details. These details are moved to rl directory.

With such decoupling, one can create a new RL algorithm inheriting the RLAlgorithmBase class, without changing the policy* class.

This PR introduces RAM-efficient seq buffer in buffers/seq_replay_buffer_efficient.py, which stores observations only once, unlike the vanilla seq buffer in buffers/seq_replay_buffer_vanilla.py storing twice. Thus it can reduce RAM roughly by 2x, especially useful for large observation space such as pixel inputs.

I test the correctness of this buffer in __main__, if you are interested, please run

python buffers/seq_replay_buffer_efficient.py

to double check.

To use it, set train: buffer_type: "seq_efficient" in config file. The default buffer type is still the vanilla one, to keep consistent with the paper's results. However, we recommend to use the efficient one, especially for atari games.