Multi Task RL Baselines

Related tags

Deep Learning mtrl
Overview

CircleCI License: MIT Python 3.6+ Code style: black Zulip Chat

MTRL

Multi Task RL Algorithms

Contents

  1. Introduction

  2. Setup

  3. Usage

  4. Documentation

  5. Contributing to MTRL

  6. Community

  7. Acknowledgements

Introduction

MTRL is a library of multi-task reinforcement learning algorithms. It has two main components:

Together, these two components enable use of MTRL across different environments and setups.

List of publications & submissions using MTRL (please create a pull request to add the missing entries):

License

Citing MTRL

If you use MTRL in your research, please use the following BibTeX entry:

@Misc{Sodhani2021MTRL,
  author =       {Shagun Sodhani and Amy Zhang},
  title =        {MTRL - Multi Task RL Algorithms},
  howpublished = {Github},
  year =         {2021},
  url =          {https://github.com/facebookresearch/mtrl}
}

Setup

  • Clone the repository: git clone [email protected]:facebookresearch/mtrl.git.

  • Install dependencies: pip install -r requirements/dev.txt

Usage

  • MTRL supports 8 different multi-task RL algorithms as described here.

  • MTRL supports multi-task environments using MTEnv. These environments include MetaWorld and multi-task variants of DMControl Suite

  • Refer the tutorial to get started with MTRL.

Documentation

https://mtrl.readthedocs.io

Contributing to MTRL

There are several ways to contribute to MTRL.

  1. Use MTRL in your research.

  2. Contribute a new algorithm. We currently support 8 multi-task RL algorithms and are looking forward to adding more environments.

  3. Check out the good-first-issues on GitHub and contribute to fixing those issues.

  4. Check out additional details here.

Community

Ask questions in the chat or github issues:

Acknowledgements

  • Our implementation of SAC is inspired by Denis Yarats' implementation of SAC.
  • Project file pre-commit, mypy config, towncrier config, circleci etc are based on same files from Hydra.
Comments
  • The centroid and star-shaped structure of DisTraL

    The centroid and star-shaped structure of DisTraL

    Description

    In 2.3 Policy Gradient and a Better Parameterization of 'Distral: Robust Multitask Reinforcement Learning', the author argues that the centroid and star-shaped structure of DisTraL is good for learning a better distilled policy. However, the explanation is too simple. Could you know the advantages of the centroid and star-shaped structure and explain them in detail? I have tried to contact the author of the paper, but I haven't received a reply. So I come to consult you.

    opened by ZJP-c4cld 5
  • Could not load experiment/metaworld. Available options: mtrl

    Could not load experiment/metaworld. Available options: mtrl

    config/config.yaml as :

    defaults:
      - setup: metaworld
      - experiment: ${defaults.0.setup}
      - agent: sac
      - agent/components: ${defaults.0.setup}_actor
      - agent/components: ${defaults.0.setup}_critic
      - agent/components: ${defaults.0.setup}_encoder
      - agent/components: ${defaults.0.setup}_transition_model
      # - agent/components: task_to_encoder_cluster
      - agent/components: ${defaults.0.setup}_mask
      - agent/components: ${defaults.0.setup}_multitask
      - agent/components: ${defaults.0.setup}_transition_model
      - agent/components: ${defaults.0.setup}_gradnorm
      - agent/components: ${defaults.0.setup}_task_to_encoder_cluster
      - agent/optimizers: ${defaults.0.setup}_actor
      - agent/optimizers: ${defaults.0.setup}_alpha
      - agent/optimizers: ${defaults.0.setup}_critic
      - agent/optimizers: ${defaults.0.setup}_decoder
      - agent/optimizers: ${defaults.0.setup}_encoder
      - env: ???
      - replay_buffer: mtrl
      - logger: mtrl
      - metrics: ${defaults.0.setup}
      - logbook: mtrl
    

    The error is shown in the screenshot. I'm new for the hydra and I just want to run a sample demo (metaworld and dm_control env are ok) log

    opened by emigmo 4
  • Can you provide the MetaWorld version used in CARE?

    Can you provide the MetaWorld version used in CARE?

    Hi @shagunsodhani ! As mentioned in #10 , Meta-World is under activate development, so it would be great if you can provide the version of Meta-World used in the CARE paper. Maybe you can specify which commit of Meta-World you used?

    opened by NagisaZj 3
  • Task version issue while running CARE baseline

    Task version issue while running CARE baseline

    Description

    When I try to run the CARE baseline, the code automatically takes version 2 of each task. Since the embeddings are available for version 1 of the tasks, it leads to an error. However, a simple hard replacement of "2" with "1" inside the code helps fix the issue. But I am not sure why version 2 of every task is being invoked by default, and if it is the correct way to fix the issue.

    How to reproduce

    Just running the CARE baseline leads to this issue.

    Stack trace / error message

    hydra.errors.HydraException: Error calling 'mtrl.experiment.metaworld.Experiment' : Error calling 'mtrl.agent.sac.Agent' : Error calling 'mtrl.agent.components.task_encoder.TaskEncoder' : 'reach-v2
    

    System information

    • MTRL Version : 1.0.0
    • Python version : 3.8.8

    Any other information

    A quick fix for this problem is changing https://github.com/facebookresearch/mtrl/blob/main/mtrl/agent/components/task_encoder.py#L42-L44 to

    pretrained_embedding = torch.Tensor(
         [metadata[task[:-1]+"1"] for task in ordered_task_list]
    )
    

    Any leads in this regard would be very helpful. Thank you!

    opened by americast 2
  • Some questions when I run distral code

    Some questions when I run distral code

    Description

    What issue are you facing?

    When I run the following code which you mention in https://mtrl.readthedocs.io/en/latest/pages/tutorials/baseline.html, error occurs.

    MUJOCO_GL="osmesa" LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/nvidia-opengl/:$LD_LIBRARY_PATH PYTHONPATH=. python3 -u main.py
    setup=hipbmdp
    env=dmcontrol-finger-spin-distribution-v0
    agent=distral
    setup.seed=1
    agent.distral_alpha=1.0
    agent.distral_beta=1.0
    replay_buffer.batch_size=256

    How to reproduce

    Add a minimal example to reproduce the issue. MUJOCO_GL="osmesa" LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/nvidia-opengl/:$LD_LIBRARY_PATH PYTHONPATH=. python3 -u main.py
    setup=hipbmdp
    env=dmcontrol-finger-spin-distribution-v0
    agent=distral
    setup.seed=1
    agent.distral_alpha=1.0
    agent.distral_beta=1.0
    replay_buffer.batch_size=256

    Stack trace / error message

    Paste the stack trace/error message to Gist and paste the link here.

    The link of error message https://gist.github.com/ZJP-c4cld/a0e052ee2d4ca8c88044ca7062fc3ab2

    System information

    • MTRL Version : The output of "pip list" in mtrl conda environment is as follows:
    • absl-py 0.12.0 aiohttp 3.7.4.post0 aiohttp-cors 0.7.0 aioredis 1.3.1 alabaster 0.7.12 antlr4-python3-runtime 4.8 anyio 3.1.0 apipkg 1.5 appdirs 1.4.4 argcomplete 1.12.3 argon2-cffi 20.1.0 astor 0.8.1 async-generator 1.10 async-timeout 3.0.1 attrs 21.2.0 Babel 2.9.1 backcall 0.2.0 beautifulsoup4 4.9.3 black 20.8b1 bleach 3.3.0 blessings 1.7 Brotli 1.0.9 cached-property 1.5.2 cachetools 4.2.2 certifi 2020.12.5 cffi 1.14.5 cfgv 3.3.0 chardet 4.0.0 click 8.0.0 cloudpickle 1.2.2 colorama 0.4.4 colorful 0.5.4 colorlog 4.8.0 contextvars 2.4 cycler 0.10.0 Cython 0.29.23 dataclasses 0.8 decorator 5.0.9 defusedxml 0.7.1 distlib 0.3.2 dm-tree 0.1.6 docopt 0.6.2 docutils 0.17.1 entrypoints 0.3 execnet 1.8.1 fasteners 0.16 filelock 3.0.12 flake8 3.8.4 flake8-bugbear 20.11.1 flake8-comprehensions 3.3.1 flake8-docstrings 1.5.0 Flask 2.0.1 Flask-Compress 1.9.0 flatbuffers 2.0 funcsigs 1.0.2 future 0.18.2 gast 0.4.0 glfw 2.1.0 google 3.0.0 google-api-core 1.27.0 google-auth 1.30.0 google-auth-oauthlib 0.4.4 googleapis-common-protos 1.53.0 gpustat 0.6.0 grpcio 1.37.1 gym 0.18.3 h5py 3.1.0 highway-env 1.0.dev0 hiplot 0.1.22 hiredis 2.0.0 Hydra 2.5 hydra-core 1.0.5 hydra-submitit-launcher 1.0.1 identify 2.2.6 idna 2.10 idna-ssl 1.1.0 imageio 2.9.0 imageio-ffmpeg 0.4.3 imagesize 1.2.0 immutables 0.15 importlib-metadata 4.0.1 importlib-resources 5.1.4 incremental 21.3.0 iniconfig 1.1.1 ipykernel 5.5.5 ipython 7.16.1 ipython-genutils 0.2.0 ipywidgets 7.6.3 isort 5.7.0 itsdangerous 2.0.1 jedi 0.18.0 Jinja2 3.0.1 joblib 1.0.1 json5 0.9.5 jsonschema 3.2.0 jupyter 1.0.0 jupyter-client 6.1.12 jupyter-console 6.4.0 jupyter-contrib-core 0.3.3 jupyter-core 4.7.1 jupyter-nbextensions-configurator 0.4.1 jupyter-server 1.8.0 jupyterlab 3.0.3 jupyterlab-pygments 0.1.2 jupyterlab-server 2.5.2 jupyterlab-widgets 1.0.0 Keras-Applications 1.0.8 Keras-Preprocessing 1.1.2 kiwisolver 1.3.1 latexcodec 2.0.1 llvmlite 0.36.0 lockfile 0.12.2 lz4 3.1.3 Markdown 3.3.4 MarkupSafe 2.0.1 matplotlib 3.3.3 mccabe 0.6.1 mistune 0.8.4 mllogger 0.7 msgpack 1.0.2 mtenv 1.0 /media/dl_e/17EF48822800F82E/zjp/code/procgen/mtrl/src/mtenv mujoco-py 2.0.2.13 multidict 5.1.0 mypy 0.790 mypy-extensions 0.4.3 nbclassic 0.3.1 nbclient 0.5.3 nbconvert 6.0.7 nbformat 5.1.3 nest-asyncio 1.5.1 nodeenv 1.6.0 notebook 6.4.0 nox 2020.12.31 numba 0.53.1 numpy 1.19.5 nvidia-ml-py3 7.352.0 oauthlib 3.1.0 omegaconf 2.0.6 opencensus 0.7.13 opencensus-context 0.1.2 opencv-python 4.5.1.48 packaging 20.9 pandas 1.1.5 pandocfilters 1.4.3 parso 0.8.2 pathspec 0.8.1 pexpect 4.8.0 pickleshare 0.7.5 Pillow 7.1.0 pip 21.1.1 pluggy 0.13.1 pockets 0.9.1 pre-commit 2.9.3 prometheus-client 0.10.1 prompt-toolkit 3.0.18 protobuf 3.17.0 psutil 5.8.0 ptyprocess 0.7.0 py 1.10.0 py-spy 0.3.7 pyasn1 0.4.8 pyasn1-modules 0.2.8 pybtex 0.24.0 pybtex-docutils 1.0.0 pycodestyle 2.6.0 pycparser 2.20 pydocstyle 6.1.1 pyflakes 2.2.0 pygame 1.9.6 pyglet 1.5.0 Pygments 2.9.0 pymongo 3.11.2 pynndescent 0.5.2 pyparsing 2.4.7 pyrsistent 0.17.3 pytest 6.2.1 pytest-forked 1.3.0 pytest-xdist 2.2.0 python-dateutil 2.8.1 pytz 2020.1 PyYAML 5.4.1 pyzmq 22.1.0 qtconsole 5.1.0 QtPy 1.9.0 ray 1.0.1.post1 redis 3.4.1 regex 2021.4.4 requests 2.25.1 requests-oauthlib 1.3.0 rl-agents 1.0.dev0 rsa 4.7.2 scikit-learn 0.24.2 scipy 1.5.4 seaborn 0.11.0 Send2Trash 1.5.0 setproctitle 1.2.2 setuptools 51.1.1 six 1.16.0 sniffio 1.2.0 snowballstemmer 2.1.0 soupsieve 2.2.1 Sphinx 3.4.3 sphinx-autodoc-annotation 1.0.post1 sphinx-copybutton 0.3.1 sphinx-rtd-theme 0.5.1 sphinxcontrib-applehelp 1.0.2 sphinxcontrib-bibtex 2.1.3 sphinxcontrib-devhelp 1.0.2 sphinxcontrib-htmlhelp 2.0.0 sphinxcontrib-jsmath 1.0.1 sphinxcontrib-napoleon 0.7 sphinxcontrib-qthelp 1.0.3 sphinxcontrib-serializinghtml 1.1.5 submitit 1.3.3 svgwrite 1.4.1 tabulate 0.8.9 tensorboard 2.4.0 tensorboard-plugin-wit 1.8.0 tensorboardX 2.1 tensorflow 1.12.0 termcolor 1.1.0 terminado 0.10.0 testpath 0.5.0 threadpoolctl 2.1.0 toml 0.10.2 torch 1.7.1+cu110 torchvision 0.8.2+cu110 tornado 6.1 towncrier 19.2.0 traitlets 4.3.3 Tree 0.2.4 typed-ast 1.4.3 typing 3.7.4.3 typing-extensions 3.7.4.3 umap-learn 0.5.1 urllib3 1.26.4 virtualenv 20.4.7 wcwidth 0.2.5 webencodings 0.5.1 websocket-client 1.0.1 Werkzeug 2.0.1 wheel 0.36.2 widgetsnbextension 3.5.1 yamllint 1.25.0 yarl 1.6.3 zipp 3.4.1
    • MTRL environment Name : mtrl
    • Python version : 3.6.5

    Any other information

    Add any other information here.

    opened by ZJP-c4cld 2
  • materials.xml is missing

    materials.xml is missing

    Description

    Hi, I meet with some missing file problem when I try to run exps with mtrl.

    After installation according to the instructions, when I try to run deepMDP or HiP-BMDP algorithms, there is an error about the following XML file is missing:

    /home/myname/anaconda3/envs/test/lib/python3.8/site-packages/local_dm_control_suite/./common/materials.xml
    

    I wonder if I miss some packages which should be installed before running experiments?

    How to reproduce

    create a new python env with anaconda:

    conda create -n test python==3.8
    

    run the following commands to install mtrl/mtenv:

    git clone https://github.com/facebookresearch/mtrl.git
    cd mtrl
    pip install -r requirements/dev.txt
    pip install mtenv
    pip install git+https://github.com/facebookresearch/mtenv.git@main#egg="mtenv[metaworld]"
    pip install git+https://github.com/facebookresearch/mtenv.git@main#egg="mtenv[hipbmdp]"
    pip install dm_control
    pip install --upgrade numpy
    

    and use the following to start the experiments (according to https://mtrl.readthedocs.io/en/latest/pages/tutorials/baseline.html):

    MUJOCO_GL="osmesa" LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/nvidia-opengl/:$LD_LIBRARY_PATH PYTHONPATH=. python3 -u main.py \
    setup=hipbmdp \
    env=dmcontrol-finger-spin-distribution-v0 \
    agent=hipbmdp \
    setup.seed=1 \
    replay_buffer.batch_size=256
    

    Error Messages:

    torch version = 1.7.1+cu110
    /home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/gym/logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
      warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
    [2021-04-08 03:35:38,823][absl][INFO] - MUJOCO_GL=osmesa, attempting to import specified OpenGL backend.
    [2021-04-08 03:35:38,830][OpenGL.acceleratesupport][INFO] - No OpenGL_accelerate module loaded: No module named 'OpenGL_accelerate'
    [2021-04-08 03:35:39,000][absl][INFO] - MuJoCo library version is: 200
    Traceback (most recent call last):
      File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/utils.py", line 66, in call
        return _call_callable(type_or_callable, config, *args, **kwargs)
      File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/_internal/utils.py", line 510, in _call_callable
        return fn(*args, **final_kwargs)
      File "/home/v-jiahuang/test_mtrl/mtrl/mtrl/env/builder.py", line 46, in build_dmcontrol_vec_env
        env = VecEnv(env_metadata=env_metadata, env_fns=funcs_to_make_envs, context="spawn")
      File "/home/v-jiahuang/test_mtrl/mtrl/mtrl/env/vec_env.py", line 22, in __init__
        super().__init__(
      File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/gym/vector/async_vector_env.py", line 77, in __init__
        dummy_env = env_fns[0]()
      File "/home/v-jiahuang/test_mtrl/mtrl/mtrl/env/builder.py", line 32, in _func
        return mtenv.make(
      File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/mtenv/envs/registration.py", line 78, in make
        env = mtenv_registry.make(id, **kwargs)
      File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/gym/envs/registration.py", line 90, in make
        env = spec.make(**kwargs)
      File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/gym/envs/registration.py", line 60, in make
        env = cls(**_kwargs)
      File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/mtenv/envs/hipbmdp/env.py", line 78, in build
        mtenv = MultiEnvWrapper(
      File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/mtenv/envs/shared/wrappers/multienv.py", line 44, in __init__
        self._envs[initial_task_state] = funcs_to_make_envs[initial_task_state]()
      File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/mtenv/envs/hipbmdp/env.py", line 58, in _func
        return dmc_env.build_dmc_env(
      File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/mtenv/envs/hipbmdp/dmc_env.py", line 96, in build_dmc_env
        env = _build_env(
      File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/mtenv/envs/hipbmdp/dmc_env.py", line 56, in _build_env
        return gym.make(env_id)
      File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/gym/envs/registration.py", line 145, in make
        return registry.make(id, **kwargs)
      File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/gym/envs/registration.py", line 90, in make
        env = spec.make(**kwargs)
      File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/gym/envs/registration.py", line 59, in make
        cls = load(self.entry_point)
      File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/gym/envs/registration.py", line 18, in load
        mod = importlib.import_module(mod_name)
      File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/importlib/__init__.py", line 127, in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
      File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
      File "<frozen importlib._bootstrap>", line 991, in _find_and_load
      File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 783, in exec_module
      File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
      File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/mtenv/envs/hipbmdp/wrappers/dmc_wrapper.py", line 10, in <module>
        import local_dm_control_suite as local_dmc_suite
      File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/local_dm_control_suite/__init__.py", line 27, in <module>
        from . import (
      File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/local_dm_control_suite/acrobot.py", line 27, in <module>
        from . import common
      File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/local_dm_control_suite/common/__init__.py", line 33, in <module>
        ASSETS = {
      File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/local_dm_control_suite/common/__init__.py", line 34, in <dictcomp>
        filename: resources.GetResource(os.path.join(_SUITE_DIR, filename))
      File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/dm_control/utils/io.py", line 22, in GetResource
        with open(name, mode=mode) as f:
    FileNotFoundError: [Errno 2] No such file or directory: '/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/local_dm_control_suite/./common/materials.xml'
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/utils.py", line 63, in call
        return _instantiate_class(type_or_callable, config, *args, **kwargs)
      File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/_internal/utils.py", line 500, in _instantiate_class
        return clazz(*args, **final_kwargs)
      File "/home/v-jiahuang/test_mtrl/mtrl/mtrl/experiment/dmcontrol.py", line 23, in __init__
        super().__init__(config, experiment_id)
      File "/home/v-jiahuang/test_mtrl/mtrl/mtrl/experiment/multitask.py", line 25, in __init__
        super().__init__(config, experiment_id)
      File "/home/v-jiahuang/test_mtrl/mtrl/mtrl/experiment/experiment.py", line 33, in __init__
        self.envs, self.env_metadata = self.build_envs()
      File "/home/v-jiahuang/test_mtrl/mtrl/mtrl/experiment/multitask.py", line 43, in build_envs
        envs[mode] = hydra.utils.instantiate(
      File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/utils.py", line 70, in call
        raise HydraException(f"Error calling '{cls}' : {e}") from e
    hydra.errors.HydraException: Error calling 'mtrl.env.builder.build_dmcontrol_vec_env' : [Errno 2] No such file or directory: '/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/local_dm_control_suite/./common/materials.xml'
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/_internal/utils.py", line 198, in run_and_report
        return func()
      File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/_internal/utils.py", line 347, in <lambda>
        lambda: hydra.run(
      File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 107, in run
        return run_job(
      File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/core/utils.py", line 128, in run_job
        ret.return_value = task_function(task_cfg)
      File "main.py", line 15, in launch
        return run(config)
      File "/home/v-jiahuang/test_mtrl/mtrl/mtrl/app/run.py", line 35, in run
        experiment_utils.prepare_and_run(config=config)
      File "/home/v-jiahuang/test_mtrl/mtrl/mtrl/experiment/utils.py", line 23, in prepare_and_run
        experiment = hydra.utils.instantiate(
      File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/utils.py", line 70, in call
        raise HydraException(f"Error calling '{cls}' : {e}") from e
    hydra.errors.HydraException: Error calling 'mtrl.experiment.dmcontrol.Experiment' : Error calling 'mtrl.env.builder.build_dmcontrol_vec_env' : [Errno 2] No such file or directory: '/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/local_dm_control_suite/./common/materials.xml'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "main.py", line 19, in <module>
        launch()
      File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/main.py", line 32, in decorated_main
        _run_hydra(
      File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/_internal/utils.py", line 346, in _run_hydra
        run_and_report(
      File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/_internal/utils.py", line 237, in run_and_report
        assert mdl is not None
    AssertionError
    

    System information

    • MTRL Version : commit eea3c99cc116e0fadc41815d0e7823349fcc0bf4
    • MTRL environment Name : dmcontrol-finger-spin-distribution-v0
    • python=3.8

    Any other information

    Btw, when I install mtrl, there is another error/warning:

     FileNotFoundError: [Errno 2] No such file or directory: 'requirements/main.txt'
    

    and indeed I can not find main.txt in requirements, but the installation is not interrupted so I guess maybe it's not an important problem.

    opened by jiaweihhuang 2
  • Error running SAC+FILM and CARE, Configuration file seems missing

    Error running SAC+FILM and CARE, Configuration file seems missing

    Description

    I am trying to run SAC+FILM and CARE according to the commands in the documentation, but it seems configuration files are missing.

    How to reproduce

    Run the SAC+FILM and CARE commands according to the documentation:

    #film
    CUDA_VISIBLE_DEVICES=4 OPENBLAS_NUM_THREADS=4 PYTHONPATH=. python3 -u main.py \
    setup=metaworld \
    env=metaworld-mt10 \
    agent=state_sac \
    experiment.num_eval_episodes=1 \
    experiment.num_train_steps=2000000 \
    setup.seed=4 \
    replay_buffer.batch_size=1280 \
    agent.multitask.num_envs=10 \
    agent.multitask.should_use_disentangled_alpha=True \
    agent.multitask.should_use_task_encoder=True \
    agent.encoder.type_to_select=film \
    agent.multitask.should_use_multi_head_policy=False \
    agent.multitask.task_encoder_cfg.model_cfg.pretrained_embedding_cfg.should_use=True \
    agent.multitask.task_encoder_cfg.model_cfg.output_dim=6
    
    #care
    CUDA_VISIBLE_DEVICES=5 OPENBLAS_NUM_THREADS=4 PYTHONPATH=. python3 -u main.py \
    setup=metaworld \
    env=metaworld-mt10 \
    agent=state_sac \
    experiment.num_eval_episodes=1 \
    experiment.num_train_steps=2000000 \
    setup.seed=5 \
    replay_buffer.batch_size=1280 \
    agent.multitask.num_envs=10 \
    agent.multitask.should_use_disentangled_alpha=True \
    agent.multitask.should_use_task_encoder=True \
    agent.encoder.type_to_select=moe \
    agent.multitask.should_use_multi_head_policy=False \
    agent.encoder.moe.task_id_to_encoder_id_cfg.mode=attention \
    agent.encoder.moe.num_experts=4 \
    agent.multitask.task_encoder_cfg.model_cfg.pretrained_embedding_cfg.should_use=True
    

    Stack trace / error message For SAC+FILM:

    Traceback (most recent call last):
      File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/utils.py", line 63, in call
        return _instantiate_class(type_or_callable, config, *args, **kwargs)
      File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/_internal/utils.py", line 500, in _instantiate_class
        return clazz(*args, **final_kwargs)
      File "/data3/zj/mtrl/mtrl/agent/components/task_encoder.py", line 39, in __init__
        with open(pretrained_embedding_cfg.path_to_load_from) as f:
    FileNotFoundError: [Errno 2] No such file or directory: '/private/home/sodhani/projects/mtrl/metadata/task_embedding/roberta_small/metaworld-mt10.json'
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/utils.py", line 63, in call
        return _instantiate_class(type_or_callable, config, *args, **kwargs)
      File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/_internal/utils.py", line 500, in _instantiate_class
        return clazz(*args, **final_kwargs)
      File "/data3/zj/mtrl/mtrl/agent/sac.py", line 116, in __init__
        self.multitask_cfg.task_encoder_cfg.model_cfg,
      File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/utils.py", line 70, in call
        raise HydraException(f"Error calling '{cls}' : {e}") from e
    hydra.errors.HydraException: Error calling 'mtrl.agent.components.task_encoder.TaskEncoder' : [Errno 2] No such file or directory: '/private/home/sodhani/projects/mtrl/metadata/task_embedding/roberta_small/metaworld-mt10.json'
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/utils.py", line 63, in call
        return _instantiate_class(type_or_callable, config, *args, **kwargs)
      File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/_internal/utils.py", line 500, in _instantiate_class
        return clazz(*args, **final_kwargs)
      File "/data3/zj/mtrl/mtrl/experiment/metaworld.py", line 20, in __init__
        super().__init__(config, experiment_id)
      File "/data3/zj/mtrl/mtrl/experiment/multitask.py", line 25, in __init__
        super().__init__(config, experiment_id)
      File "/data3/zj/mtrl/mtrl/experiment/experiment.py", line 72, in __init__
        device=self.device,
      File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/utils.py", line 70, in call
        raise HydraException(f"Error calling '{cls}' : {e}") from e
    hydra.errors.HydraException: Error calling 'mtrl.agent.sac.Agent' : Error calling 'mtrl.agent.components.task_encoder.TaskEncoder' : [Errno 2] No such file or directory: '/private/home/sodhani/projects/mtrl/metadata/task_embedding/roberta_small/metaworld-mt10.json'
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/_internal/utils.py", line 198, in run_and_report
        return func()
      File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/_internal/utils.py", line 350, in <lambda>
        overrides=args.overrides,
      File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/_internal/hydra.py", line 112, in run
        configure_logging=with_log_configuration,
      File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/core/utils.py", line 128, in run_job
        ret.return_value = task_function(task_cfg)
      File "main.py", line 15, in launch
        return run(config)
      File "/data3/zj/mtrl/mtrl/app/run.py", line 35, in run
        experiment_utils.prepare_and_run(config=config)
      File "/data3/zj/mtrl/mtrl/experiment/utils.py", line 24, in prepare_and_run
        config.experiment.builder, config
      File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/utils.py", line 70, in call
        raise HydraException(f"Error calling '{cls}' : {e}") from e
    hydra.errors.HydraException: Error calling 'mtrl.experiment.metaworld.Experiment' : Error calling 'mtrl.agent.sac.Agent' : Error calling 'mtrl.agent.components.task_encoder.TaskEncoder' : [Errno 2] No such file or directory: '/private/home/sodhani/projects/mtrl/metadata/task_embedding/roberta_small/metaworld-mt10.json'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "main.py", line 19, in <module>
        launch()
      File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/main.py", line 37, in decorated_main
        strict=strict,
      File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/_internal/utils.py", line 347, in _run_hydra
        lambda: hydra.run(
      File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/_internal/utils.py", line 237, in run_and_report
        assert mdl is not None
    AssertionError
    
    

    For CARE:

    Traceback (most recent call last):
      File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/utils.py", line 63, in call
        return _instantiate_class(type_or_callable, config, *args, **kwargs)
      File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/_internal/utils.py", line 500, in _instantiate_class
        return clazz(*args, **final_kwargs)
      File "/data3/zj/mtrl/mtrl/agent/components/task_encoder.py", line 39, in __init__
        with open(pretrained_embedding_cfg.path_to_load_from) as f:
    FileNotFoundError: [Errno 2] No such file or directory: '/private/home/sodhani/projects/mtrl/metadata/task_embedding/roberta_small/metaworld-mt10.json'
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/utils.py", line 63, in call
        return _instantiate_class(type_or_callable, config, *args, **kwargs)
      File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/_internal/utils.py", line 500, in _instantiate_class
        return clazz(*args, **final_kwargs)
      File "/data3/zj/mtrl/mtrl/agent/sac.py", line 116, in __init__
        self.multitask_cfg.task_encoder_cfg.model_cfg,
      File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/utils.py", line 70, in call
        raise HydraException(f"Error calling '{cls}' : {e}") from e
    hydra.errors.HydraException: Error calling 'mtrl.agent.components.task_encoder.TaskEncoder' : [Errno 2] No such file or directory: '/private/home/sodhani/projects/mtrl/metadata/task_embedding/roberta_small/metaworld-mt10.json'
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/utils.py", line 63, in call
        return _instantiate_class(type_or_callable, config, *args, **kwargs)
      File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/_internal/utils.py", line 500, in _instantiate_class
        return clazz(*args, **final_kwargs)
      File "/data3/zj/mtrl/mtrl/experiment/metaworld.py", line 20, in __init__
        super().__init__(config, experiment_id)
      File "/data3/zj/mtrl/mtrl/experiment/multitask.py", line 25, in __init__
        super().__init__(config, experiment_id)
      File "/data3/zj/mtrl/mtrl/experiment/experiment.py", line 72, in __init__
        device=self.device,
      File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/utils.py", line 70, in call
        raise HydraException(f"Error calling '{cls}' : {e}") from e
    hydra.errors.HydraException: Error calling 'mtrl.agent.sac.Agent' : Error calling 'mtrl.agent.components.task_encoder.TaskEncoder' : [Errno 2] No such file or directory: '/private/home/sodhani/projects/mtrl/metadata/task_embedding/roberta_small/metaworld-mt10.json'
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/_internal/utils.py", line 198, in run_and_report
        return func()
      File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/_internal/utils.py", line 350, in <lambda>
        overrides=args.overrides,
      File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/_internal/hydra.py", line 112, in run
        configure_logging=with_log_configuration,
      File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/core/utils.py", line 128, in run_job
        ret.return_value = task_function(task_cfg)
      File "main.py", line 15, in launch
        return run(config)
      File "/data3/zj/mtrl/mtrl/app/run.py", line 35, in run
        experiment_utils.prepare_and_run(config=config)
      File "/data3/zj/mtrl/mtrl/experiment/utils.py", line 24, in prepare_and_run
        config.experiment.builder, config
      File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/utils.py", line 70, in call
        raise HydraException(f"Error calling '{cls}' : {e}") from e
    hydra.errors.HydraException: Error calling 'mtrl.experiment.metaworld.Experiment' : Error calling 'mtrl.agent.sac.Agent' : Error calling 'mtrl.agent.components.task_encoder.TaskEncoder' : [Errno 2] No such file or directory: '/private/home/sodhani/projects/mtrl/metadata/task_embedding/roberta_small/metaworld-mt10.json'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "main.py", line 19, in <module>
        launch()
      File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/main.py", line 37, in decorated_main
        strict=strict,
      File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/_internal/utils.py", line 347, in _run_hydra
        lambda: hydra.run(
      File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/_internal/utils.py", line 237, in run_and_report
        assert mdl is not None
    AssertionError
    
    

    System information

    • MTRL Version : Latest commit on github (e282c3f5a205970b4ad7d1c1ebae7aa9b4d56218)
    • MTRL environment Name : context
    • Python version : 3.6.13

    Any other information

    Add any other information here.

    opened by NagisaZj 1
  • Run the code in the shell, but not work in the pycharm

    Run the code in the shell, but not work in the pycharm

    Description

    I can run this code in the shell command,

    but with the same remote settings and Conda environment in the Pycharm, it always occurs:

    raise HydraException(f"Error calling '{cls}' : {e}") from e hydra.errors.HydraException: Error calling 'mtrl.experiment.metaworld.Experiment' : Error calling 'metaworld.MT10' : Error loading module 'metaworld.MT10'

    How to reproduce

    python main.py setup=metaworld env=metaworld-mt10 agent=state_sac experiment.num_eval_episodes=1 experiment.num_train_steps=2000000 setup.seed=1 replay_buffer.batch_size=1280 agent.multitask.num_envs=10 agent.multitask.should_use_disentangled_alpha=True agent.multitask.should_use_task_encoder=True agent.encoder.type_to_select=film agent.multitask.should_use_multi_head_policy=False agent.multitask.task_encoder_cfg.model_cfg.pretrained_embedding_cfg.should_use=True agent.multitask.task_encoder_cfg.model_cfg.output_dim=6
    

    System information

    • MTRL Version : 1
    • Python version : 3.6.13
    opened by Amanda-1997 1
  • When I run CARE, it occurs `ModuleNotFoundError: No module named 'ml_logger.logbook'`

    When I run CARE, it occurs `ModuleNotFoundError: No module named 'ml_logger.logbook'`

    Description

    ModuleNotFoundError: No module named 'ml_logger.logbook'

    System information

    • ml_logger version : 0.7.17
    • Python version : 3.6.13

    Any other information

    running CARE

    opened by Amanda-1997 1
  • Fix typos related to HiP-BMDP

    Fix typos related to HiP-BMDP

    Thank you for contributing to MTRL.

    Proposed Change

    Fixing typos related to HiP-BMDP

    Have you read the Contributing Guidelines?

    Yes

    Related Issues/PRs

    #4

    CLA Signed 
    opened by shagunsodhani 1
  • Some questions related to the implementation of HiP-BMDP algorithm

    Some questions related to the implementation of HiP-BMDP algorithm

    Description

    Hi, it's not an issue about running the code, but some possible typos in the implementation of HiP-BMDP. (so I just ignore the format...)

    (1) About the configuration file ./config/agent/components/hipbmdp_multitask.yaml

    The default setting is should_use_task_encoder: False. I wonder that, if we want to include environment encoding as a part of the input of the transition model (as Figure 2 in the paper), shall we set it to be True?

    Besides, I tried to run the code after setting it to be true, to avoid dimension mismatch error when running the code, I also need to set should_condition_encoder_on_task_info and should_concatenate_task_info_with_encoder to be True in that file. I would like to know that if my current setting is correct for HiP-BMDP algorithm.

    (2) maybe a typo in ./mtrl/agent/hipbmdp.py

    In line 110, the second argument of function get_task_encoding() is mode, but when this function is called in ./mtrl/agent/sac.py Line 200, the second argument is specified as modes, and it will cause an error when running the code. I guess it's just a typo and I can run the code after fixing it.

    Could you help to fix these issues, or please let me know if I misunderstand something? Thanks!

    opened by jiaweihhuang 1
  • Mutiple values for agent/optimizers.

    Mutiple values for agent/optimizers.

    Description

    I run the code as the doc described

    PYTHONPATH=. python3 -u main.py \
    setup=metaworld \
    env=metaworld-mt10 \
    agent=state_sac \
    experiment.num_eval_episodes=1 \
    experiment.num_train_steps=2000000 \
    setup.seed=1 \
    replay_buffer.batch_size=1280 \
    agent.multitask.num_envs=10 \
    agent.multitask.should_use_disentangled_alpha=True \
    agent.encoder.type_to_select=identity \
    agent.multitask.should_use_multi_head_policy=False \
    agent.multitask.actor_cfg.should_condition_model_on_task_info=False \
    agent.multitask.actor_cfg.should_condition_encoder_on_task_info=True \
    agent.multitask.actor_cfg.should_concatenate_task_info_with_encoder=True
    

    And it shows that

    Multiple values for agent/optimizers. To override a value use 'override agent/optimizers: metaworld_encoder' Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

    How can I solve this problem? Thanks~

    How to reproduce

    Add a minimal example to reproduce the issue.

    Stack trace / error message

    Paste the stack trace/error message to Gist and paste the link here.

    System information

    • MTRL Version :
    • MTRL environment Name :
    • Python version :

    Any other information

    Add any other information here.

    opened by chenci107 0
  • Support for metaworld-v2

    Support for metaworld-v2

    Description

    When can the mtrl repo support metaworld-v2, which has been released for an whole year and should be considered as the new standard environment for multi-task rl?

    @shagunsodhani

    How to reproduce

    Add a minimal example to reproduce the issue.

    Stack trace / error message

    Paste the stack trace/error message to Gist and paste the link here.

    System information

    • MTRL Version :
    • MTRL environment Name :
    • Python version :

    Any other information

    Add any other information here.

    opened by waterhorse1 0
  • Can not run the demo !!!! How to config a minimal runnable case.

    Can not run the demo !!!! How to config a minimal runnable case.

    Description

    • setup: metaworld
    • experiment: metaworld
    • agent: sac
    • agent/components: ${defaults.0.setup}_actor #- agent/components: ${defaults.0.setup}_critic #- agent/components: ${defaults.0.setup}_encoder #- agent/components: ${defaults.0.setup}_transition_model #- agent/components: task_to_encoder_cluster #- agent/components: ${defaults.0.setup}_mask #- agent/components: ${defaults.0.setup}_multitask #- agent/components: ${defaults.0.setup}_transition_model #- agent/components: ${defaults.0.setup}_gradnorm #- agent/components: ${defaults.0.setup}_task_to_encoder_cluster
    • agent/optimizers: ${defaults.0.setup}_actor #- agent/optimizers: ${defaults.0.setup}_alpha #- agent/optimizers: ${defaults.0.setup}_critic #- agent/optimizers: ${defaults.0.setup}_decoder #- agent/optimizers: ${defaults.0.setup}_encoder
    • env: metaworld-mt10
    • replay_buffer: mtrl
    • logger: mtrl
    • metrics: ${defaults.0.setup}
    • logbook: mtrl

    Running log:

    setup=metaworld agent=state_sac env=metaworld-mt10 agent.multitask.num_envs=10 agent.multitask.should_use_disentangled_alpha=True /home/zoulixin/miniconda3/lib/python3.9/site-packages/gym/envs/registration.py:440: UserWarning: WARN: The registry.env_specs property along with EnvSpecTree is deprecated. Please use registry directly as a dictionary instead. logger.warn( Traceback (most recent call last): File "/home/zoulixin/data/keras-mmoe/CAGrad-main/mtrl/temp/mtrl-main/main.py", line 7, in from mtrl.app.run import run File "/home/zoulixin/data/keras-mmoe/CAGrad-main/mtrl/temp/mtrl-main/mtrl/app/run.py", line 7, in from mtrl.experiment import utils as experiment_utils File "/home/zoulixin/data/keras-mmoe/CAGrad-main/mtrl/temp/mtrl-main/mtrl/experiment/utils.py", line 11, in from mtrl.experiment.metaworld import Experiment File "/home/zoulixin/data/keras-mmoe/CAGrad-main/mtrl/temp/mtrl-main/mtrl/experiment/metaworld.py", line 10, in from mtrl.env import builder as env_builder File "/home/zoulixin/data/keras-mmoe/CAGrad-main/mtrl/temp/mtrl-main/mtrl/env/builder.py", line 5, in import mtenv File "/home/zoulixin/miniconda3/lib/python3.9/site-packages/mtenv/init.py", line 5, in from mtenv.envs.registration import make # noqa: F401 File "/home/zoulixin/miniconda3/lib/python3.9/site-packages/mtenv/envs/init.py", line 10, in register( File "/home/zoulixin/miniconda3/lib/python3.9/site-packages/mtenv/envs/registration.py", line 74, in register return mtenv_registry.register(id, **kwargs) File "/home/zoulixin/miniconda3/lib/python3.9/site-packages/mtenv/envs/registration.py", line 66, in register self.env_specs[id] = MultitaskEnvSpec(id, **kwargs) File "/home/zoulixin/miniconda3/lib/python3.9/site-packages/mtenv/envs/registration.py", line 41, in init super().init( File "", line 12, in init AttributeError: can't set attribute

    How to reproduce

    Add a minimal example to reproduce the issue.

    Stack trace / error message

    Paste the stack trace/error message to Gist and paste the link here.

    System information

    • MTRL Version :
    • MTRL environment Name :
    • Python version :

    Any other information

    Add any other information here.

    opened by zoulixin93 0
  • AttributeError: can't set attribute

    AttributeError: can't set attribute

    Description

    I was following docs, and when I tried to use command

    PYTHONPATH=. python3 -u main.py \
    setup=metaworld \
    agent=state_sac \
    env=metaworld-mt10 \
    agent.multitask.num_envs=10 \
    agent.multitask.should_use_disentangled_alpha=True
    

    I kept getting AttributeError: can't set attribute error.

    How to reproduce

    PYTHONPATH=. python3 -u main.py \
    setup=metaworld \
    agent=state_sac \
    env=metaworld-mt10 \
    agent.multitask.num_envs=10 \
    agent.multitask.should_use_disentangled_alpha=True
    

    and the full error message is here

    System information

    • MTRL Version : e282c3f5a205970b4ad7d1c1ebae7aa9b4d56218 commit
    • MTRL environment Name : metaworld-mt10
    • Python version : 3.7

    Any other information

    I use mujoco-py==2.0.2.0 and af8417bfc82a3e249b4b02156518d775f29eb289 commit for the MetaWorld environments, and I have also run through following two test codes to ensure my mujoco-py and metaworld was installed successfully. mujoco-py test:

    import mujoco_py
    import os
    mj_path, _ = mujoco_py.utils.discover_mujoco()
    xml_path = os.path.join(mj_path, 'model', 'humanoid.xml')
    model = mujoco_py.load_model_from_path(xml_path)
    sim = mujoco_py.MjSim(model)
    
    print(sim.data.qpos)
    

    metaworld test:

    import metaworld
    import random
    
    mt10 = metaworld.MT10() # Construct the benchmark, sampling tasks
    
    training_envs = []
    for name, env_cls in mt10.train_classes.items():
      env = env_cls()
      task = random.choice([task for task in mt10.train_tasks
                            if task.env_name == name])
      env.set_task(task)
      training_envs.append(env)
    
    for env in training_envs:
      obs = env.reset()  # Reset environment
      a = env.action_space.sample()  # Sample an action
      obs, reward, done, info = env.step(a)  # Step the environoment with the sampled random action
    

    Also, the full step I create my environment is here

    opened by Tonylin1998 2
  • Failed to run the demo metaworld-mt10

    Failed to run the demo metaworld-mt10

    Description

    An error was encountered after running the sample code for a while, I use such command line settings: PYTHONPATH=. python3 -u main.py
    setup=metaworld
    agent=state_sac
    env=metaworld-mt10
    agent.multitask.num_envs=10
    agent.multitask.should_use_disentangled_alpha=True

    How to reproduce

    Error in call to target 'mtrl.experiment.metaworld.Experiment': ConfigAttributeError('Missing key encoder\n full_key: agent.encoder\n object_type=dict') full_key: experiment.builder

    The above exception was the direct cause of the following exception:

    File "C:\Users\Tz0942\Downloads\mtrl-main\mtrl\experiment\metaworld.py", line 20, in init super().init(config, experiment_id) File "C:\Users\Tz0942\Downloads\mtrl-main\mtrl\experiment\multitask.py", line 25, in init super().init(config, experiment_id) File "C:\Users\Tz0942\Downloads\mtrl-main\mtrl\experiment\experiment.py", line 63, in init self.config = prepare_config(config=self.config, env_metadata=self.env_metadata) File "C:\Users\Tz0942\Downloads\mtrl-main\mtrl\experiment\experiment.py", line 187, in prepare_config if key in config.agent.encoder:

    The above exception was the direct cause of the following exception:

    System information

    • MTRL Version : e282c3f5a205970b4ad7d1c1ebae7aa9b4d56218
    • MTRL environment Name : meatworld-mt10
    • Python version : python 3.8
    • metaworld commit: af8417bfc82a3e249b4b02156518d775f29eb289

    Any other information

    I have to modify some configuration items in config.yaml can be annotated before I run main.py successfully: config.yaml like this: defaults: - setup: metaworld - experiment: ${defaults.0.setup} - agent: sac - agent/components: ${defaults.0.setup}_actor #- agent/components: ${defaults.0.setup}_critic #- agent/components: ${defaults.0.setup}_encoder #- agent/components: ${defaults.0.setup}_transition_model #- agent/components: task_to_encoder_cluster #- agent/components: ${defaults.0.setup}_mask #- agent/components: ${defaults.0.setup}_multitask #- agent/components: ${defaults.0.setup}_transition_model #- agent/components: ${defaults.0.setup}_gradnorm #- agent/components: ${defaults.0.setup}_task_to_encoder_cluster - agent/optimizers: ${defaults.0.setup}_actor #- agent/optimizers: ${defaults.0.setup}_alpha #- agent/optimizers: ${defaults.0.setup}_critic #- agent/optimizers: ${defaults.0.setup}_decoder #- agent/optimizers: ${defaults.0.setup}_encoder - env: ??? - replay_buffer: mtrl - logger: mtrl - metrics: ${defaults.0.setup} - logbook: mtrl

    opened by Zhulongtao6 0
  • Fail to get MT10 upper bound

    Fail to get MT10 upper bound

    Description

    image

    we have successfully trained some algorithm on MT10. However,when we train a sac agent on MT1 to get "One SAC agent per task(upper bound)", it always fails due to critic loss is to high( >1e8), and the success rate is near 0%. Is there any special config for MT1?

    How to reproduce

    we use the following config:

    setup=metaworld
    env=metaworld-mt1
    agent=state_sac
    experiment.num_eval_episodes=1
    experiment.num_train_steps=2000000
    setup.seed=10
    replay_buffer.batch_size=1280
    agent.multitask.num_envs=1
    agent.multitask.should_use_disentangled_alpha=False
    agent.encoder.type_to_select=identity
    agent.multitask.should_use_multi_head_policy=False
    agent.multitask.actor_cfg.should_condition_model_on_task_info=False
    agent.multitask.actor_cfg.should_condition_encoder_on_task_info=True
    agent.multitask.actor_cfg.should_concatenate_task_info_with_encoder=True

    image we change the default task_name in the function get_list_of_func_to_make_envs() ( src/mtenv/mtenv/envs/metaworld/env.py) to control the task uesd in MT1.

    System information

    • MTRL Version : latest
    • Metaword Version : af8417bfc82a3e249b4b02156518d775f29eb289

    .

    opened by niiceMing 0
Owner
Facebook Research
Facebook Research
Baselines for TrajNet++

TrajNet++ : The Trajectory Forecasting Framework PyTorch implementation of Human Trajectory Forecasting in Crowds: A Deep Learning Perspective TrajNet

VITA lab at EPFL 183 Jan 5, 2023
PyTorch version of Stable Baselines, reliable implementations of reinforcement learning algorithms.

PyTorch version of Stable Baselines, reliable implementations of reinforcement learning algorithms.

DLR-RM 4.7k Jan 1, 2023
Multi-task Multi-agent Soft Actor Critic for SMAC

Multi-task Multi-agent Soft Actor Critic for SMAC Overview The CARE formulti-task: Multi-Task Reinforcement Learning with Context-based Representation

RuanJingqing 8 Sep 30, 2022
Multi-Object Tracking in Satellite Videos with Graph-Based Multi-Task Modeling

TGraM Multi-Object Tracking in Satellite Videos with Graph-Based Multi-Task Modeling, Qibin He, Xian Sun, Zhiyuan Yan, Beibei Li, Kun Fu Abstract Rece

Qibin He 6 Nov 25, 2022
Code and pre-trained models for MultiMAE: Multi-modal Multi-task Masked Autoencoders

MultiMAE: Multi-modal Multi-task Masked Autoencoders Roman Bachmann*, David Mizrahi*, Andrei Atanov, Amir Zamir Website | arXiv | BibTeX Official PyTo

Visual Intelligence & Learning Lab, Swiss Federal Institute of Technology (EPFL) 385 Jan 6, 2023
Code of U2Fusion: a unified unsupervised image fusion network for multiple image fusion tasks, including multi-modal, multi-exposure and multi-focus image fusion.

U2Fusion Code of U2Fusion: a unified unsupervised image fusion network for multiple image fusion tasks, including multi-modal (VIS-IR, medical), multi

Han Xu 129 Dec 11, 2022
A list of multi-task learning papers and projects.

This page contains a list of papers on multi-task learning for computer vision. Please create a pull request if you wish to add anything. If you are interested, consider reading our recent survey paper.

svandenh 297 Dec 17, 2022
A list of multi-task learning papers and projects.

A list of multi-task learning papers and projects.

svandenh 84 Apr 27, 2021
AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning

AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning (NeurIPS 2020) Introduction AdaShare is a novel and differentiable approach fo

null 94 Dec 22, 2022
RoboDesk A Multi-Task Reinforcement Learning Benchmark

RoboDesk A Multi-Task Reinforcement Learning Benchmark If you find this open source release useful, please reference in your paper: @misc{kannan2021ro

Google Research 66 Oct 7, 2022
Multi-Task Temporal Shift Attention Networks for On-Device Contactless Vitals Measurement (NeurIPS 2020)

MTTS-CAN: Multi-Task Temporal Shift Attention Networks for On-Device Contactless Vitals Measurement Paper Xin Liu, Josh Fromm, Shwetak Patel, Daniel M

Xin Liu 106 Dec 30, 2022
Implementation of PyTorch-based multi-task pre-trained models

mtdp Library containing implementation related to the research paper "Multi-task pre-training of deep neural networks for digital pathology" (Mormont

Romain Mormont 27 Oct 14, 2022
Code for the ICML 2021 paper "Bridging Multi-Task Learning and Meta-Learning: Towards Efficient Training and Effective Adaptation", Haoxiang Wang, Han Zhao, Bo Li.

Bridging Multi-Task Learning and Meta-Learning Code for the ICML 2021 paper "Bridging Multi-Task Learning and Meta-Learning: Towards Efficient Trainin

AI Secure 57 Dec 15, 2022
Multi Task Vision and Language

12-in-1: Multi-Task Vision and Language Representation Learning Please cite the following if you use this code. Code and pre-trained models for 12-in-

Facebook Research 712 Dec 19, 2022
Exploring Relational Context for Multi-Task Dense Prediction [ICCV 2021]

Adaptive Task-Relational Context (ATRC) This repository provides source code for the ICCV 2021 paper Exploring Relational Context for Multi-Task Dense

David Brüggemann 35 Dec 5, 2022
This code uses generative adversarial networks to generate diverse task allocation plans for Multi-agent teams.

Mutli-agent task allocation This code uses generative adversarial networks to generate diverse task allocation plans for Multi-agent teams. To change

Biorobotics Lab 5 Oct 12, 2022
Face Detection and Alignment using Multi-task Cascaded Convolutional Networks (MTCNN)

Face-Detection-with-MTCNN Face detection is a computer vision problem that involves finding faces in photos. It is a trivial problem for humans to sol

Chetan Hirapara 3 Oct 7, 2022
Multi-task yolov5 with detection and segmentation based on yolov5

YOLOv5DS Multi-task yolov5 with detection and segmentation based on yolov5(branch v6.0) decoupled head anchor free segmentation head README中文 Ablation

null 150 Dec 30, 2022
Multi-task Learning of Order-Consistent Causal Graphs (NeuRIPs 2021)

Multi-task Learning of Order-Consistent Causal Graphs (NeuRIPs 2021) Authors: Xinshi Chen, Haoran Sun, Caleb Ellington, Eric Xing, Le Song Link to pap

Xinshi Chen 2 Dec 20, 2021