Code for "Offline Meta-Reinforcement Learning with Advantage Weighting" [ICML 2021]

Eric Mitchell

Last update: Jan 1, 2023

Related tags

Deep Learning macaw

Overview

Offline Meta-Reinforcement Learning with Advantage Weighting (MACAW)

MACAW code used for the experiments in the ICML 2021 paper.

Installing the environment

# Install Python 3.7.9 if necessary
$ pyenv install 3.7.9
$ pyenv shell 3.7.9

$ python --version
Python 3.7.9

$ python -m venv env
$ source env/bin/activate
$ pip install -r requirements.txt

Downloading the data

The offline data used for MACAW can be found here. Download it and use the default name (macaw_offline_data) for the folder where the four data directories are stored. gDrive might be useful here if downloading from the Google Drive GUI is not an option.

Running MACAW 🦜

Run offline meta-training with periodic online evaluations with any of the scripts in scripts/. e.g.

$ . scripts/macaw_dir.sh # MACAW training on Cheetah-Direction (Figure 1)
$ . scripts/macaw_vel.sh # MACAW training on Cheetah-Velocity (Figure 1)
$ . scripts/macaw_quality_ablation.sh # Data quality ablation (Figure 5-left)
...

Outputs (tensorboard logs) will be written to the log/ directory.

Reach out!

If you're having issues with the code or data, feel free to open an issue or send me an email.

Citation

If our code or research was useful for your own work, you can cite us with the following attribution:

@InProceedings{mitchell2021offline,
    title = {Offline Meta-Reinforcement Learning with Advantage Weighting},
    author = {Mitchell, Eric and Rafailov, Rafael and Peng, Xue Bin and Levine, Sergey and Finn, Chelsea},
    booktitle = {Proceedings of the 38th International Conference on Machine Learning},
    year = {2021}
}

An interpreter for RASP as described in the ICML 2021 paper "Thinking Like Transformers"

RASP Setup Mac or Linux Run ./setup.sh . It will create a python3 virtual environment and install the dependencies for RASP. It will also try to insta

141 Jan 3, 2023

Official implementation of "SinIR: Efficient General Image Manipulation with Single Image Reconstruction" (ICML 2021)

SinIR (Official Implementation) Requirements To install requirements: pip install -r requirements.txt We used Python 3.7.4 and f-strings which are in

47 Oct 11, 2022

[ICML 2021] “ Self-Damaging Contrastive Learning”, Ziyu Jiang, Tianlong Chen, Bobak Mortazavi, Zhangyang Wang

Self-Damaging Contrastive Learning Introduction The recent breakthrough achieved by contrastive learning accelerates the pace for deploying unsupervis

51 Dec 29, 2022

[ICML 2021] "Graph Contrastive Learning Automated" by Yuning You, Tianlong Chen, Yang Shen, Zhangyang Wang

Graph Contrastive Learning Automated PyTorch implementation for Graph Contrastive Learning Automated [talk] [poster] [appendix] Yuning You, Tianlong C

80 Nov 23, 2022

How Do Adam and Training Strategies Help BNNs Optimization? In ICML 2021.

AdamBNN This is the pytorch implementation of our paper "How Do Adam and Training Strategies Help BNNs Optimization?", published in ICML 2021. In this

47 Sep 20, 2022

[ICML 2021] Break-It-Fix-It: Learning to Repair Programs from Unlabeled Data

Break-It-Fix-It: Learning to Repair Programs from Unlabeled Data This repo provides the source code & data of our paper: Break-It-Fix-It: Unsupervised

86 Nov 30, 2022

Implementation of Learning Gradient Fields for Molecular Conformation Generation (ICML 2021).

[PDF] | [Slides] The official implementation of Learning Gradient Fields for Molecular Conformation Generation (ICML 2021 Long talk) Installation Inst

117 Dec 9, 2022

Pytorch Implementation of Spiking Neural Networks Calibration, ICML 2021

SNN_Calibration Pytorch Implementation of Spiking Neural Networks Calibration, ICML 2021 Feature Comparison of SNN calibration: Features SNN Direct Tr

60 Dec 27, 2022

Implementation of Self-supervised Graph-level Representation Learning with Local and Global Structure (ICML 2021).

Self-supervised Graph-level Representation Learning with Local and Global Structure Introduction This project is an implementation of ``Self-supervise

50 Dec 9, 2022

Comments

About algorithm 1

First, thanks for sharing the codes.

In algorithm 1, for the test part (line 10 and 11). Where the task iteration is ended, so why there is still index i in the math expressions? Could you elaborate?

opened by HYDesmondLiu 1
cannot run this code

Dear mitchell,

Thanks for the sharing code. However, when I run the code, these is a error:

ModuleNotFoundError: No module named 'rand_param_envs'

This error comes from this file macaw-main/src/tp_envs/walker_rand_params_wrapper.py

It seems missing some files.

Looking forward to your reply!

Best Yiqin

opened by YiqinYang 1
Missing File: "config/alg/standard_loadvel.json" and "env/bin/activate"

When running bash ./scripts/macaw_vel.sh Error comes out that :

FileNotFoundError: [Errno 2] No such file or directory: 'config/alg/standard_loadvel.json'

./scripts/macaw_vel.sh: line 3: env/bin/activate: No such file or directory

Would you like to provide "config/alg/standard_loadvel.json"? And dose "env/bin/activate" matters?

opened by han-x 1
Metaworld offline data

Hi, I'm interested in your work and thanks for your sharing code. In the paper, you conduct ML45 experiments on metaworld benchmarks, however, the relevant offline data are not provided by you. Could you please offer the ML45 offline data? Thanks for your reply!

opened by kevin-xuan 1

Code for "Offline Meta-Reinforcement Learning with Advantage Weighting" [ICML 2021]

Related tags

Overview

Offline Meta-Reinforcement Learning with Advantage Weighting (MACAW)

Installing the environment

Downloading the data

Running MACAW 🦜

Reach out!

Citation

You might also like...

An interpreter for RASP as described in the ICML 2021 paper "Thinking Like Transformers"

Official implementation of "SinIR: Efficient General Image Manipulation with Single Image Reconstruction" (ICML 2021)

[ICML 2021] “ Self-Damaging Contrastive Learning”, Ziyu Jiang, Tianlong Chen, Bobak Mortazavi, Zhangyang Wang

[ICML 2021] "Graph Contrastive Learning Automated" by Yuning You, Tianlong Chen, Yang Shen, Zhangyang Wang

How Do Adam and Training Strategies Help BNNs Optimization? In ICML 2021.

[ICML 2021] Break-It-Fix-It: Learning to Repair Programs from Unlabeled Data

Implementation of Learning Gradient Fields for Molecular Conformation Generation (ICML 2021).

Pytorch Implementation of Spiking Neural Networks Calibration, ICML 2021

Implementation of Self-supervised Graph-level Representation Learning with Local and Global Structure (ICML 2021).

Comments

About algorithm 1

cannot run this code

Missing File: "config/alg/standard_loadvel.json" and "env/bin/activate"

Metaworld offline data

Owner

Eric Mitchell

Code for ICML 2021 paper: How could Neural Networks understand Programs?

Official Code for ICML 2021 paper "Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline"

Code for Fold2Seq paper from ICML 2021

Official code for UnICORNN (ICML 2021)

Code for the ICML 2021 paper "Bridging Multi-Task Learning and Meta-Learning: Towards Efficient Training and Effective Adaptation", Haoxiang Wang, Han Zhao, Bo Li.

Code release for "Self-Tuning for Data-Efficient Deep Learning" (ICML 2021)

Code release for the ICML 2021 paper "PixelTransformer: Sample Conditioned Signal Generation".

TensorFlow code for the neural network presented in the paper: "Structural Language Models of Code" (ICML'2020)

[ICML 2021, Long Talk] Delving into Deep Imbalanced Regression

[ICML 2021] DouZero: Mastering DouDizhu with Self-Play Deep Reinforcement Learning | 斗地主AI