Pytorch implementations of popular off-policy multi-agent reinforcement learning algorithms, including QMix, VDN, MADDPG, and MATD3.

Last update: Dec 28, 2022

Related tags

Deep Learning off-policy

Overview

Off-Policy Multi-Agent Reinforcement Learning (MARL) Algorithms

This repository contains implementations of various off-policy multi-agent reinforcement learning (MARL) algorithms.

Authors: Akash Velu and Chao Yu

Algorithms supported:

MADDPG (MLP and RNN)
MATD3 (MLP and RNN)
QMIX (MLP and RNN)
VDN (MLP and RNN)

Environments supported:

1. Usage

WARNING #1: by default all experiments assume a shared policy by all agents i.e. there is one neural network shared by all agents

WARNING #2: only QMIX and MADDPG are thoroughly tested; however,our VDN and MATD3 implementations make small modifications to QMIX and MADDPG, respectively. We display results using our implementation here.

All core code is located within the offpolicy folder. The algorithms/ subfolder contains algorithm-specific code for all methods. RMADDPG and RMATD3 refer to RNN implementationso of MADDPG and MATD3, and mQMIX and mVDN refer to MLP implementations of QMIX and VDN. We additionally support prioritized experience replay (PER).

The envs/ subfolder contains environment wrapper implementations for the MPEs and SMAC.
Code to perform training rollouts and policy updates are contained within the runner/ folder - there is a runner for each environment.
Executable scripts for training with default hyperparameters can be found in the scripts/ folder. The files are named in the following manner: train_algo_environment.sh. Within each file, the map name (in the case of SMAC and the MPEs) can be altered.
Python training scripts for each environment can be found in the scripts/train/ folder.
The config.py file contains relevant hyperparameter and env settings. Most hyperparameters are defaulted to the ones used in the paper; however, please refer to the appendix for a full list of hyperparameters used.

2. Installation

Here we give an example installation on CUDA == 10.1. For non-GPU & other CUDA version installation, please refer to the PyTorch website.

# create conda environment
conda create -n marl python==3.6.1
conda activate marl
pip install torch==1.5.1+cu101 torchvision==0.6.1+cu101 -f https://download.pytorch.org/whl/torch_stable.html

# install on-policy package
cd on-policy
pip install -e .

Even though we provide requirement.txt, it may have redundancy. We recommend that the user try to install other required packages by running the code and finding which required package hasn't installed yet.

2.1 Install StarCraftII 4.10

unzip SC2.4.10.zip
# password is iagreetotheeula
echo "export SC2PATH=~/StarCraftII/" > ~/.bashrc

download SMAC Maps, and move it to ~/StarCraftII/Maps/.
To use a stableid, copy stableid.json from https://github.com/Blizzard/s2client-proto.git to ~/StarCraftII/.

2.2 Install MPE

# install this package first
pip install seaborn

There are 3 Cooperative scenarios in MPE:

simple_spread
simple_speaker_listener, which is 'Comm' scenario in paper
simple_reference

3.Train

Here we use train_mpe_maddpg.sh as an example:

cd offpolicy/scripts
chmod +x ./train_mpe_maddpg.sh
./train_mpe_maddpg.sh

Local results are stored in subfold scripts/results. Note that we use Weights & Bias as the default visualization platform; to use Weights & Bias, please register and login to the platform first. More instructions for using Weights&Bias can be found in the official documentation. Adding the --use_wandb in command line or in the .sh file will use Tensorboard instead of Weights & Biases.

4. Results

Results for the performance of RMADDPG and QMIX on the Particle Envs and QMIX in SMAC are depicted here. These results are obtained using a normal (not prioitized) replay buffer.

Comments

Error with wandb

Hello, I have encountered some problems. I wonder if you can help me. that:Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving. wandb: Currently logged in as: zhouweiqing (use wandb login --relogin to force relogin) wandb: ERROR Error while calling W&B API: project not found (<Response [404]>) requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://api.wandb.ai/graphql i can not solve it

Originally posted by @zhouweiqing-star in https://github.com/marlbenchmark/off-policy/issues/3#issuecomment-1049665762

opened by Maxtoq 3
fix(wzl): fix vdn mixer to avoid Q value dim mismatch with reward

command

./train_mpe_vdn.sh

problem

Traceback (most recent call last): File "train/train_mpe.py", line 192, in main(sys.argv[1:]) File "train/train_mpe.py", line 177, in main total_num_steps = runner.run() File "/home/zerlinwang/Projects/off-policy/offpolicy/runner/rnn/base_runner.py", line 190, in run self.train() File "/home/zerlinwang/Projects/off-policy/offpolicy/runner/rnn/base_runner.py", line 272, in batch_train_q train_info, new_priorities, idxes = self.trainer.train_policy_on_batch(sample) File "/home/zerlinwang/Projects/off-policy/offpolicy/algorithms/qmix/qmix.py", line 164, in train_policy_on_batch Q_tot_target_seq = rewards + (1 - dones_env_batch) * self.args.gamma * next_step_Q_tot_seq RuntimeError: The size of tensor a (32) must match the size of tensor b (800) at non-singleton dimension 1

reason

next_step_Q_tot_seq dim error

solution

~~return agent_q_inps.sum(dim=-1).view(-1, 1, 1)~~ -> batch_size = agent_q_inps.size(1) return agent_q_inps.sum(dim=-1).view(-1, batch_size, 1, 1)

result

opened by zerlinwang 0
fix(wzl): change [] to nn.ModuleList in MDDPG_Critic to avoid differe…

command

sh ./train_mpe_maddpg.sh

error

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

reason

self.q_outs = [init_(nn.Linear(self.hidden_size, 1)) for _ in range(num_q_outs)] self.to(device) In this way, the to(device) won’t transfer these parameters to the desired device.

ref

https://discuss.pytorch.org/t/is-it-mandatory-to-add-modules-to-modulelist-to-access-its-parameters/81622/7

opened by zerlinwang 0
Can you open-source MASAC code base?

Hello, Thanks for open-sourcing a really good work. I was wondering if you guys can open-source the MASAC code base as it would help to understand the variations of MASAC with MADDPG. Thanks, in advance for the help.

opened by kailashg26 0
RuntimeError: CUDA error: an illegal memory access was encountered

Traceback (most recent call last): File "train_mpe.py", line 157, in main(sys.argv[1:]) File "train_mpe.py", line 147, in main total_num_steps = runner.run() File "D:\off-policy-release\offpolicy\runner\mlp\base_runner.py", line 153, in run env_info = self.collecter(explore=True, training_episode=True, warmup=False) File "D:\off-policy-release\offpolicy\runner\mlp\mpe_runner.py", line 145, in shared_collect_rollout self.train() File "D:\off-policy-release\offpolicy\runner\mlp\base_runner.py", line 189, in batch_train train_info, new_priorities, idxes = update(p_id, sample) File "D:\off-policy-release\offpolicy\algorithms\maddpg\maddpg.py", line 117, in shared_train_policy_on_batch rewards = to_torch(rewards).to(**self.tpdv).view(-1, 1) RuntimeError: CUDA error: an illegal memory access was encountered When I run the maddpg, it encount cuda error

opened by b762927 0
Some questions about the code

why did the code require only one env when using rnn policy? https://github.com/marlbenchmark/off-policy/blob/release/offpolicy/scripts/train/train_mpe.py#L154

opened by rainbow979 1

Pytorch implementations of popular off-policy multi-agent reinforcement learning algorithms, including QMix, VDN, MADDPG, and MATD3.

Related tags

Overview

Off-Policy Multi-Agent Reinforcement Learning (MARL) Algorithms

Algorithms supported:

Environments supported:

1. Usage

2. Installation

2.1 Install StarCraftII 4.10

2.2 Install MPE

3.Train

4. Results

Comments

Error with wandb

fix(wzl): fix vdn mixer to avoid Q value dim mismatch with reward

command

problem

reason

solution

result

fix(wzl): change [] to nn.ModuleList in MDDPG_Critic to avoid differe…

command

error

reason

ref

Can you open-source MASAC code base?

RuntimeError: CUDA error: an illegal memory access was encountered

Some questions about the code

Owner

A Pytorch implementation of the multi agent deep deterministic policy gradients (MADDPG) algorithm

Independent and minimal implementations of some reinforcement learning algorithms using PyTorch (including PPO, A3C, A2C, ...).

Reinforcement learning library(framework) designed for PyTorch, implements DQN, DDPG, A2C, PPO, SAC, MADDPG, A3C, APEX, IMPALA ...

Official Implementation of 'UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformers' ICLR 2021(spotlight)

DRLib：A concise deep reinforcement learning library, integrating HER and PER for almost off policy RL algos.

Implementation of Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning

Offline Multi-Agent Reinforcement Learning Implementations: Solving Overcooked Game with Data-Driven Method

Deep Reinforcement Learning by using an on-policy adaptation of Maximum a Posteriori Policy Optimization (MPO)

This project uses reinforcement learning on stock market and agent tries to learn trading. The goal is to check if the agent can learn to read tape. The project is dedicated to hero in life great Jesse Livermore.

PyTorch implementations of deep reinforcement learning algorithms and environments

PyTorch version of Stable Baselines, reliable implementations of reinforcement learning algorithms.

Scripts of Machine Learning Algorithms from Scratch. Implementations of machine learning models and algorithms using nothing but NumPy with a focus on accessibility. Aims to cover everything from basic to advance.

Weighted QMIX: Expanding Monotonic Value Function Factorisation

Learning to Communicate with Deep Multi-Agent Reinforcement Learning in PyTorch

PyTorch implementation of Advantage Actor Critic (A2C), Proximal Policy Optimization (PPO), Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation (ACKTR) and Generative Adversarial Imitation Learning (GAIL).

PyTorch implementation of Advantage Actor Critic (A2C), Proximal Policy Optimization (PPO), Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation (ACKTR) and Generative Adversarial Imitation Learning (GAIL).

A library of multi-agent reinforcement learning components and systems

Multi-agent reinforcement learning algorithm and environment

Code of U2Fusion: a unified unsupervised image fusion network for multiple image fusion tasks, including multi-modal, multi-exposure and multi-focus image fusion.