Description

In 2.3 Policy Gradient and a Better Parameterization of 'Distral: Robust Multitask Reinforcement Learning', the author argues that the centroid and star-shaped structure of DisTraL is good for learning a better distilled policy. However, the explanation is too simple. Could you know the advantages of the centroid and star-shaped structure and explain them in detail? I have tried to contact the author of the paper, but I haven't received a reply. So I come to consult you.

config/config.yaml as :

defaults:
  - setup: metaworld
  - experiment: ${defaults.0.setup}
  - agent: sac
  - agent/components: ${defaults.0.setup}_actor
  - agent/components: ${defaults.0.setup}_critic
  - agent/components: ${defaults.0.setup}_encoder
  - agent/components: ${defaults.0.setup}_transition_model
  # - agent/components: task_to_encoder_cluster
  - agent/components: ${defaults.0.setup}_mask
  - agent/components: ${defaults.0.setup}_multitask
  - agent/components: ${defaults.0.setup}_transition_model
  - agent/components: ${defaults.0.setup}_gradnorm
  - agent/components: ${defaults.0.setup}_task_to_encoder_cluster
  - agent/optimizers: ${defaults.0.setup}_actor
  - agent/optimizers: ${defaults.0.setup}_alpha
  - agent/optimizers: ${defaults.0.setup}_critic
  - agent/optimizers: ${defaults.0.setup}_decoder
  - agent/optimizers: ${defaults.0.setup}_encoder
  - env: ???
  - replay_buffer: mtrl
  - logger: mtrl
  - metrics: ${defaults.0.setup}
  - logbook: mtrl

The error is shown in the screenshot. I'm new for the hydra and I just want to run a sample demo (metaworld and dm_control env are ok)

Hi @shagunsodhani ! As mentioned in #10 , Meta-World is under activate development, so it would be great if you can provide the version of Meta-World used in the CARE paper. Maybe you can specify which commit of Meta-World you used?

Description

When I try to run the CARE baseline, the code automatically takes version 2 of each task. Since the embeddings are available for version 1 of the tasks, it leads to an error. However, a simple hard replacement of "2" with "1" inside the code helps fix the issue. But I am not sure why version 2 of every task is being invoked by default, and if it is the correct way to fix the issue.

How to reproduce

Just running the CARE baseline leads to this issue.

Stack trace / error message

hydra.errors.HydraException: Error calling 'mtrl.experiment.metaworld.Experiment' : Error calling 'mtrl.agent.sac.Agent' : Error calling 'mtrl.agent.components.task_encoder.TaskEncoder' : 'reach-v2

System information

• MTRL Version : 1.0.0
• Python version : 3.8.8

Any other information

A quick fix for this problem is changing https://github.com/facebookresearch/mtrl/blob/main/mtrl/agent/components/task_encoder.py#L42-L44 to

pretrained_embedding = torch.Tensor(
     [metadata[task[:-1]+"1"] for task in ordered_task_list]
)

Any leads in this regard would be very helpful. Thank you!

Description

What issue are you facing?

When I run the following code which you mention in https://mtrl.readthedocs.io/en/latest/pages/tutorials/baseline.html, error occurs.

MUJOCO_GL="osmesa" LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/nvidia-opengl/:$LD_LIBRARY_PATH PYTHONPATH=. python3 -u main.py
setup=hipbmdp
env=dmcontrol-finger-spin-distribution-v0
agent=distral
setup.seed=1
agent.distral_alpha=1.0
agent.distral_beta=1.0
replay_buffer.batch_size=256

How to reproduce

Add a minimal example to reproduce the issue. MUJOCO_GL="osmesa" LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/nvidia-opengl/:$LD_LIBRARY_PATH PYTHONPATH=. python3 -u main.py
setup=hipbmdp
env=dmcontrol-finger-spin-distribution-v0
agent=distral
setup.seed=1
agent.distral_alpha=1.0
agent.distral_beta=1.0
replay_buffer.batch_size=256

Stack trace / error message

Paste the stack trace/error message to Gist and paste the link here.

The link of error message https://gist.github.com/ZJP-c4cld/a0e052ee2d4ca8c88044ca7062fc3ab2

System information

• MTRL Version : The output of "pip list" in mtrl conda environment is as follows:
• absl-py 0.12.0 aiohttp 3.7.4.post0 aiohttp-cors 0.7.0 aioredis 1.3.1 alabaster 0.7.12 antlr4-python3-runtime 4.8 anyio 3.1.0 apipkg 1.5 appdirs 1.4.4 argcomplete 1.12.3 argon2-cffi 20.1.0 astor 0.8.1 async-generator 1.10 async-timeout 3.0.1 attrs 21.2.0 Babel 2.9.1 backcall 0.2.0 beautifulsoup4 4.9.3 black 20.8b1 bleach 3.3.0 blessings 1.7 Brotli 1.0.9 cached-property 1.5.2 cachetools 4.2.2 certifi 2020.12.5 cffi 1.14.5 cfgv 3.3.0 chardet 4.0.0 click 8.0.0 cloudpickle 1.2.2 colorama 0.4.4 colorful 0.5.4 colorlog 4.8.0 contextvars 2.4 cycler 0.10.0 Cython 0.29.23 dataclasses 0.8 decorator 5.0.9 defusedxml 0.7.1 distlib 0.3.2 dm-tree 0.1.6 docopt 0.6.2 docutils 0.17.1 entrypoints 0.3 execnet 1.8.1 fasteners 0.16 filelock 3.0.12 flake8 3.8.4 flake8-bugbear 20.11.1 flake8-comprehensions 3.3.1 flake8-docstrings 1.5.0 Flask 2.0.1 Flask-Compress 1.9.0 flatbuffers 2.0 funcsigs 1.0.2 future 0.18.2 gast 0.4.0 glfw 2.1.0 google 3.0.0 google-api-core 1.27.0 google-auth 1.30.0 google-auth-oauthlib 0.4.4 googleapis-common-protos 1.53.0 gpustat 0.6.0 grpcio 1.37.1 gym 0.18.3 h5py 3.1.0 highway-env 1.0.dev0 hiplot 0.1.22 hiredis 2.0.0 Hydra 2.5 hydra-core 1.0.5 hydra-submitit-launcher 1.0.1 identify 2.2.6 idna 2.10 idna-ssl 1.1.0 imageio 2.9.0 imageio-ffmpeg 0.4.3 imagesize 1.2.0 immutables 0.15 importlib-metadata 4.0.1 importlib-resources 5.1.4 incremental 21.3.0 iniconfig 1.1.1 ipykernel 5.5.5 ipython 7.16.1 ipython-genutils 0.2.0 ipywidgets 7.6.3 isort 5.7.0 itsdangerous 2.0.1 jedi 0.18.0 Jinja2 3.0.1 joblib 1.0.1 json5 0.9.5 jsonschema 3.2.0 jupyter 1.0.0 jupyter-client 6.1.12 jupyter-console 6.4.0 jupyter-contrib-core 0.3.3 jupyter-core 4.7.1 jupyter-nbextensions-configurator 0.4.1 jupyter-server 1.8.0 jupyterlab 3.0.3 jupyterlab-pygments 0.1.2 jupyterlab-server 2.5.2 jupyterlab-widgets 1.0.0 Keras-Applications 1.0.8 Keras-Preprocessing 1.1.2 kiwisolver 1.3.1 latexcodec 2.0.1 llvmlite 0.36.0 lockfile 0.12.2 lz4 3.1.3 Markdown 3.3.4 MarkupSafe 2.0.1 matplotlib 3.3.3 mccabe 0.6.1 mistune 0.8.4 mllogger 0.7 msgpack 1.0.2 mtenv 1.0 /media/dl_e/17EF48822800F82E/zjp/code/procgen/mtrl/src/mtenv mujoco-py 2.0.2.13 multidict 5.1.0 mypy 0.790 mypy-extensions 0.4.3 nbclassic 0.3.1 nbclient 0.5.3 nbconvert 6.0.7 nbformat 5.1.3 nest-asyncio 1.5.1 nodeenv 1.6.0 notebook 6.4.0 nox 2020.12.31 numba 0.53.1 numpy 1.19.5 nvidia-ml-py3 7.352.0 oauthlib 3.1.0 omegaconf 2.0.6 opencensus 0.7.13 opencensus-context 0.1.2 opencv-python 4.5.1.48 packaging 20.9 pandas 1.1.5 pandocfilters 1.4.3 parso 0.8.2 pathspec 0.8.1 pexpect 4.8.0 pickleshare 0.7.5 Pillow 7.1.0 pip 21.1.1 pluggy 0.13.1 pockets 0.9.1 pre-commit 2.9.3 prometheus-client 0.10.1 prompt-toolkit 3.0.18 protobuf 3.17.0 psutil 5.8.0 ptyprocess 0.7.0 py 1.10.0 py-spy 0.3.7 pyasn1 0.4.8 pyasn1-modules 0.2.8 pybtex 0.24.0 pybtex-docutils 1.0.0 pycodestyle 2.6.0 pycparser 2.20 pydocstyle 6.1.1 pyflakes 2.2.0 pygame 1.9.6 pyglet 1.5.0 Pygments 2.9.0 pymongo 3.11.2 pynndescent 0.5.2 pyparsing 2.4.7 pyrsistent 0.17.3 pytest 6.2.1 pytest-forked 1.3.0 pytest-xdist 2.2.0 python-dateutil 2.8.1 pytz 2020.1 PyYAML 5.4.1 pyzmq 22.1.0 qtconsole 5.1.0 QtPy 1.9.0 ray 1.0.1.post1 redis 3.4.1 regex 2021.4.4 requests 2.25.1 requests-oauthlib 1.3.0 rl-agents 1.0.dev0 rsa 4.7.2 scikit-learn 0.24.2 scipy 1.5.4 seaborn 0.11.0 Send2Trash 1.5.0 setproctitle 1.2.2 setuptools 51.1.1 six 1.16.0 sniffio 1.2.0 snowballstemmer 2.1.0 soupsieve 2.2.1 Sphinx 3.4.3 sphinx-autodoc-annotation 1.0.post1 sphinx-copybutton 0.3.1 sphinx-rtd-theme 0.5.1 sphinxcontrib-applehelp 1.0.2 sphinxcontrib-bibtex 2.1.3 sphinxcontrib-devhelp 1.0.2 sphinxcontrib-htmlhelp 2.0.0 sphinxcontrib-jsmath 1.0.1 sphinxcontrib-napoleon 0.7 sphinxcontrib-qthelp 1.0.3 sphinxcontrib-serializinghtml 1.1.5 submitit 1.3.3 svgwrite 1.4.1 tabulate 0.8.9 tensorboard 2.4.0 tensorboard-plugin-wit 1.8.0 tensorboardX 2.1 tensorflow 1.12.0 termcolor 1.1.0 terminado 0.10.0 testpath 0.5.0 threadpoolctl 2.1.0 toml 0.10.2 torch 1.7.1+cu110 torchvision 0.8.2+cu110 tornado 6.1 towncrier 19.2.0 traitlets 4.3.3 Tree 0.2.4 typed-ast 1.4.3 typing 3.7.4.3 typing-extensions 3.7.4.3 umap-learn 0.5.1 urllib3 1.26.4 virtualenv 20.4.7 wcwidth 0.2.5 webencodings 0.5.1 websocket-client 1.0.1 Werkzeug 2.0.1 wheel 0.36.2 widgetsnbextension 3.5.1 yamllint 1.25.0 yarl 1.6.3 zipp 3.4.1
• MTRL environment Name : mtrl
• Python version : 3.6.5

Any other information

Add any other information here.

Description

Hi, I meet with some missing file problem when I try to run exps with mtrl.

After installation according to the instructions, when I try to run deepMDP or HiP-BMDP algorithms, there is an error about the following XML file is missing:

/home/myname/anaconda3/envs/test/lib/python3.8/site-packages/local_dm_control_suite/./common/materials.xml

I wonder if I miss some packages which should be installed before running experiments?

How to reproduce

create a new python env with anaconda:

conda create -n test python==3.8

run the following commands to install mtrl/mtenv:

git clone https://github.com/facebookresearch/mtrl.git
cd mtrl
pip install -r requirements/dev.txt
pip install mtenv
pip install git+https://github.com/facebookresearch/mtenv.git@main#egg="mtenv[metaworld]"
pip install git+https://github.com/facebookresearch/mtenv.git@main#egg="mtenv[hipbmdp]"
pip install dm_control
pip install --upgrade numpy

and use the following to start the experiments (according to https://mtrl.readthedocs.io/en/latest/pages/tutorials/baseline.html):

MUJOCO_GL="osmesa" LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/nvidia-opengl/:$LD_LIBRARY_PATH PYTHONPATH=. python3 -u main.py \
setup=hipbmdp \
env=dmcontrol-finger-spin-distribution-v0 \
agent=hipbmdp \
setup.seed=1 \
replay_buffer.batch_size=256

Error Messages:

torch version = 1.7.1+cu110
/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/gym/logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
[2021-04-08 03:35:38,823][absl][INFO] - MUJOCO_GL=osmesa, attempting to import specified OpenGL backend.
[2021-04-08 03:35:38,830][OpenGL.acceleratesupport][INFO] - No OpenGL_accelerate module loaded: No module named 'OpenGL_accelerate'
[2021-04-08 03:35:39,000][absl][INFO] - MuJoCo library version is: 200
Traceback (most recent call last):
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/utils.py", line 66, in call
    return _call_callable(type_or_callable, config, *args, **kwargs)
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/_internal/utils.py", line 510, in _call_callable
    return fn(*args, **final_kwargs)
  File "/home/v-jiahuang/test_mtrl/mtrl/mtrl/env/builder.py", line 46, in build_dmcontrol_vec_env
    env = VecEnv(env_metadata=env_metadata, env_fns=funcs_to_make_envs, context="spawn")
  File "/home/v-jiahuang/test_mtrl/mtrl/mtrl/env/vec_env.py", line 22, in __init__
    super().__init__(
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/gym/vector/async_vector_env.py", line 77, in __init__
    dummy_env = env_fns[0]()
  File "/home/v-jiahuang/test_mtrl/mtrl/mtrl/env/builder.py", line 32, in _func
    return mtenv.make(
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/mtenv/envs/registration.py", line 78, in make
    env = mtenv_registry.make(id, **kwargs)
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/gym/envs/registration.py", line 90, in make
    env = spec.make(**kwargs)
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/gym/envs/registration.py", line 60, in make
    env = cls(**_kwargs)
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/mtenv/envs/hipbmdp/env.py", line 78, in build
    mtenv = MultiEnvWrapper(
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/mtenv/envs/shared/wrappers/multienv.py", line 44, in __init__
    self._envs[initial_task_state] = funcs_to_make_envs[initial_task_state]()
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/mtenv/envs/hipbmdp/env.py", line 58, in _func
    return dmc_env.build_dmc_env(
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/mtenv/envs/hipbmdp/dmc_env.py", line 96, in build_dmc_env
    env = _build_env(
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/mtenv/envs/hipbmdp/dmc_env.py", line 56, in _build_env
    return gym.make(env_id)
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/gym/envs/registration.py", line 145, in make
    return registry.make(id, **kwargs)
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/gym/envs/registration.py", line 90, in make
    env = spec.make(**kwargs)
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/gym/envs/registration.py", line 59, in make
    cls = load(self.entry_point)
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/gym/envs/registration.py", line 18, in load
    mod = importlib.import_module(mod_name)
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/mtenv/envs/hipbmdp/wrappers/dmc_wrapper.py", line 10, in <module>
    import local_dm_control_suite as local_dmc_suite
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/local_dm_control_suite/__init__.py", line 27, in <module>
    from . import (
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/local_dm_control_suite/acrobot.py", line 27, in <module>
    from . import common
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/local_dm_control_suite/common/__init__.py", line 33, in <module>
    ASSETS = {
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/local_dm_control_suite/common/__init__.py", line 34, in <dictcomp>
    filename: resources.GetResource(os.path.join(_SUITE_DIR, filename))
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/dm_control/utils/io.py", line 22, in GetResource
    with open(name, mode=mode) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/local_dm_control_suite/./common/materials.xml'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/utils.py", line 63, in call
    return _instantiate_class(type_or_callable, config, *args, **kwargs)
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/_internal/utils.py", line 500, in _instantiate_class
    return clazz(*args, **final_kwargs)
  File "/home/v-jiahuang/test_mtrl/mtrl/mtrl/experiment/dmcontrol.py", line 23, in __init__
    super().__init__(config, experiment_id)
  File "/home/v-jiahuang/test_mtrl/mtrl/mtrl/experiment/multitask.py", line 25, in __init__
    super().__init__(config, experiment_id)
  File "/home/v-jiahuang/test_mtrl/mtrl/mtrl/experiment/experiment.py", line 33, in __init__
    self.envs, self.env_metadata = self.build_envs()
  File "/home/v-jiahuang/test_mtrl/mtrl/mtrl/experiment/multitask.py", line 43, in build_envs
    envs[mode] = hydra.utils.instantiate(
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/utils.py", line 70, in call
    raise HydraException(f"Error calling '{cls}' : {e}") from e
hydra.errors.HydraException: Error calling 'mtrl.env.builder.build_dmcontrol_vec_env' : [Errno 2] No such file or directory: '/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/local_dm_control_suite/./common/materials.xml'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/_internal/utils.py", line 198, in run_and_report
    return func()
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/_internal/utils.py", line 347, in <lambda>
    lambda: hydra.run(
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 107, in run
    return run_job(
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/core/utils.py", line 128, in run_job
    ret.return_value = task_function(task_cfg)
  File "main.py", line 15, in launch
    return run(config)
  File "/home/v-jiahuang/test_mtrl/mtrl/mtrl/app/run.py", line 35, in run
    experiment_utils.prepare_and_run(config=config)
  File "/home/v-jiahuang/test_mtrl/mtrl/mtrl/experiment/utils.py", line 23, in prepare_and_run
    experiment = hydra.utils.instantiate(
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/utils.py", line 70, in call
    raise HydraException(f"Error calling '{cls}' : {e}") from e
hydra.errors.HydraException: Error calling 'mtrl.experiment.dmcontrol.Experiment' : Error calling 'mtrl.env.builder.build_dmcontrol_vec_env' : [Errno 2] No such file or directory: '/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/local_dm_control_suite/./common/materials.xml'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 19, in <module>
    launch()
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/main.py", line 32, in decorated_main
    _run_hydra(
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/_internal/utils.py", line 346, in _run_hydra
    run_and_report(
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/_internal/utils.py", line 237, in run_and_report
    assert mdl is not None
AssertionError

System information

• MTRL Version : commit eea3c99cc116e0fadc41815d0e7823349fcc0bf4
• MTRL environment Name : dmcontrol-finger-spin-distribution-v0
• python=3.8

Any other information

Btw, when I install mtrl, there is another error/warning:

 FileNotFoundError: [Errno 2] No such file or directory: 'requirements/main.txt'

and indeed I can not find main.txt in requirements, but the installation is not interrupted so I guess maybe it's not an important problem.

Description

I am trying to run SAC+FILM and CARE according to the commands in the documentation, but it seems configuration files are missing.

How to reproduce

Run the SAC+FILM and CARE commands according to the documentation:

#film
CUDA_VISIBLE_DEVICES=4 OPENBLAS_NUM_THREADS=4 PYTHONPATH=. python3 -u main.py \
setup=metaworld \
env=metaworld-mt10 \
agent=state_sac \
experiment.num_eval_episodes=1 \
experiment.num_train_steps=2000000 \
setup.seed=4 \
replay_buffer.batch_size=1280 \
agent.multitask.num_envs=10 \
agent.multitask.should_use_disentangled_alpha=True \
agent.multitask.should_use_task_encoder=True \
agent.encoder.type_to_select=film \
agent.multitask.should_use_multi_head_policy=False \
agent.multitask.task_encoder_cfg.model_cfg.pretrained_embedding_cfg.should_use=True \
agent.multitask.task_encoder_cfg.model_cfg.output_dim=6

#care
CUDA_VISIBLE_DEVICES=5 OPENBLAS_NUM_THREADS=4 PYTHONPATH=. python3 -u main.py \
setup=metaworld \
env=metaworld-mt10 \
agent=state_sac \
experiment.num_eval_episodes=1 \
experiment.num_train_steps=2000000 \
setup.seed=5 \
replay_buffer.batch_size=1280 \
agent.multitask.num_envs=10 \
agent.multitask.should_use_disentangled_alpha=True \
agent.multitask.should_use_task_encoder=True \
agent.encoder.type_to_select=moe \
agent.multitask.should_use_multi_head_policy=False \
agent.encoder.moe.task_id_to_encoder_id_cfg.mode=attention \
agent.encoder.moe.num_experts=4 \
agent.multitask.task_encoder_cfg.model_cfg.pretrained_embedding_cfg.should_use=True

Stack trace / error message For SAC+FILM:

Traceback (most recent call last):
  File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/utils.py", line 63, in call
    return _instantiate_class(type_or_callable, config, *args, **kwargs)
  File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/_internal/utils.py", line 500, in _instantiate_class
    return clazz(*args, **final_kwargs)
  File "/data3/zj/mtrl/mtrl/agent/components/task_encoder.py", line 39, in __init__
    with open(pretrained_embedding_cfg.path_to_load_from) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/private/home/sodhani/projects/mtrl/metadata/task_embedding/roberta_small/metaworld-mt10.json'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/utils.py", line 63, in call
    return _instantiate_class(type_or_callable, config, *args, **kwargs)
  File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/_internal/utils.py", line 500, in _instantiate_class
    return clazz(*args, **final_kwargs)
  File "/data3/zj/mtrl/mtrl/agent/sac.py", line 116, in __init__
    self.multitask_cfg.task_encoder_cfg.model_cfg,
  File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/utils.py", line 70, in call
    raise HydraException(f"Error calling '{cls}' : {e}") from e
hydra.errors.HydraException: Error calling 'mtrl.agent.components.task_encoder.TaskEncoder' : [Errno 2] No such file or directory: '/private/home/sodhani/projects/mtrl/metadata/task_embedding/roberta_small/metaworld-mt10.json'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/utils.py", line 63, in call
    return _instantiate_class(type_or_callable, config, *args, **kwargs)
  File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/_internal/utils.py", line 500, in _instantiate_class
    return clazz(*args, **final_kwargs)
  File "/data3/zj/mtrl/mtrl/experiment/metaworld.py", line 20, in __init__
    super().__init__(config, experiment_id)
  File "/data3/zj/mtrl/mtrl/experiment/multitask.py", line 25, in __init__
    super().__init__(config, experiment_id)
  File "/data3/zj/mtrl/mtrl/experiment/experiment.py", line 72, in __init__
    device=self.device,
  File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/utils.py", line 70, in call
    raise HydraException(f"Error calling '{cls}' : {e}") from e
hydra.errors.HydraException: Error calling 'mtrl.agent.sac.Agent' : Error calling 'mtrl.agent.components.task_encoder.TaskEncoder' : [Errno 2] No such file or directory: '/private/home/sodhani/projects/mtrl/metadata/task_embedding/roberta_small/metaworld-mt10.json'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/_internal/utils.py", line 198, in run_and_report
    return func()
  File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/_internal/utils.py", line 350, in <lambda>
    overrides=args.overrides,
  File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/_internal/hydra.py", line 112, in run
    configure_logging=with_log_configuration,
  File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/core/utils.py", line 128, in run_job
    ret.return_value = task_function(task_cfg)
  File "main.py", line 15, in launch
    return run(config)
  File "/data3/zj/mtrl/mtrl/app/run.py", line 35, in run
    experiment_utils.prepare_and_run(config=config)
  File "/data3/zj/mtrl/mtrl/experiment/utils.py", line 24, in prepare_and_run
    config.experiment.builder, config
  File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/utils.py", line 70, in call
    raise HydraException(f"Error calling '{cls}' : {e}") from e
hydra.errors.HydraException: Error calling 'mtrl.experiment.metaworld.Experiment' : Error calling 'mtrl.agent.sac.Agent' : Error calling 'mtrl.agent.components.task_encoder.TaskEncoder' : [Errno 2] No such file or directory: '/private/home/sodhani/projects/mtrl/metadata/task_embedding/roberta_small/metaworld-mt10.json'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 19, in <module>
    launch()
  File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/main.py", line 37, in decorated_main
    strict=strict,
  File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/_internal/utils.py", line 347, in _run_hydra
    lambda: hydra.run(
  File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/_internal/utils.py", line 237, in run_and_report
    assert mdl is not None
AssertionError

For CARE:

Traceback (most recent call last):
  File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/utils.py", line 63, in call
    return _instantiate_class(type_or_callable, config, *args, **kwargs)
  File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/_internal/utils.py", line 500, in _instantiate_class
    return clazz(*args, **final_kwargs)
  File "/data3/zj/mtrl/mtrl/agent/components/task_encoder.py", line 39, in __init__
    with open(pretrained_embedding_cfg.path_to_load_from) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/private/home/sodhani/projects/mtrl/metadata/task_embedding/roberta_small/metaworld-mt10.json'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/utils.py", line 63, in call
    return _instantiate_class(type_or_callable, config, *args, **kwargs)
  File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/_internal/utils.py", line 500, in _instantiate_class
    return clazz(*args, **final_kwargs)
  File "/data3/zj/mtrl/mtrl/agent/sac.py", line 116, in __init__
    self.multitask_cfg.task_encoder_cfg.model_cfg,
  File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/utils.py", line 70, in call
    raise HydraException(f"Error calling '{cls}' : {e}") from e
hydra.errors.HydraException: Error calling 'mtrl.agent.components.task_encoder.TaskEncoder' : [Errno 2] No such file or directory: '/private/home/sodhani/projects/mtrl/metadata/task_embedding/roberta_small/metaworld-mt10.json'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/utils.py", line 63, in call
    return _instantiate_class(type_or_callable, config, *args, **kwargs)
  File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/_internal/utils.py", line 500, in _instantiate_class
    return clazz(*args, **final_kwargs)
  File "/data3/zj/mtrl/mtrl/experiment/metaworld.py", line 20, in __init__
    super().__init__(config, experiment_id)
  File "/data3/zj/mtrl/mtrl/experiment/multitask.py", line 25, in __init__
    super().__init__(config, experiment_id)
  File "/data3/zj/mtrl/mtrl/experiment/experiment.py", line 72, in __init__
    device=self.device,
  File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/utils.py", line 70, in call
    raise HydraException(f"Error calling '{cls}' : {e}") from e
hydra.errors.HydraException: Error calling 'mtrl.agent.sac.Agent' : Error calling 'mtrl.agent.components.task_encoder.TaskEncoder' : [Errno 2] No such file or directory: '/private/home/sodhani/projects/mtrl/metadata/task_embedding/roberta_small/metaworld-mt10.json'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/_internal/utils.py", line 198, in run_and_report
    return func()
  File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/_internal/utils.py", line 350, in <lambda>
    overrides=args.overrides,
  File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/_internal/hydra.py", line 112, in run
    configure_logging=with_log_configuration,
  File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/core/utils.py", line 128, in run_job
    ret.return_value = task_function(task_cfg)
  File "main.py", line 15, in launch
    return run(config)
  File "/data3/zj/mtrl/mtrl/app/run.py", line 35, in run
    experiment_utils.prepare_and_run(config=config)
  File "/data3/zj/mtrl/mtrl/experiment/utils.py", line 24, in prepare_and_run
    config.experiment.builder, config
  File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/utils.py", line 70, in call
    raise HydraException(f"Error calling '{cls}' : {e}") from e
hydra.errors.HydraException: Error calling 'mtrl.experiment.metaworld.Experiment' : Error calling 'mtrl.agent.sac.Agent' : Error calling 'mtrl.agent.components.task_encoder.TaskEncoder' : [Errno 2] No such file or directory: '/private/home/sodhani/projects/mtrl/metadata/task_embedding/roberta_small/metaworld-mt10.json'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 19, in <module>
    launch()
  File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/main.py", line 37, in decorated_main
    strict=strict,
  File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/_internal/utils.py", line 347, in _run_hydra
    lambda: hydra.run(
  File "/home/zj/anaconda3/envs/context/lib/python3.6/site-packages/hydra/_internal/utils.py", line 237, in run_and_report
    assert mdl is not None
AssertionError

System information

• MTRL Version : Latest commit on github (e282c3f5a205970b4ad7d1c1ebae7aa9b4d56218)
• MTRL environment Name : context
• Python version : 3.6.13

Any other information

Add any other information here.

Description

I can run this code in the shell command,

but with the same remote settings and Conda environment in the Pycharm, it always occurs:

raise HydraException(f"Error calling '{cls}' : {e}") from e hydra.errors.HydraException: Error calling 'mtrl.experiment.metaworld.Experiment' : Error calling 'metaworld.MT10' : Error loading module 'metaworld.MT10'

How to reproduce

python main.py setup=metaworld env=metaworld-mt10 agent=state_sac experiment.num_eval_episodes=1 experiment.num_train_steps=2000000 setup.seed=1 replay_buffer.batch_size=1280 agent.multitask.num_envs=10 agent.multitask.should_use_disentangled_alpha=True agent.multitask.should_use_task_encoder=True agent.encoder.type_to_select=film agent.multitask.should_use_multi_head_policy=False agent.multitask.task_encoder_cfg.model_cfg.pretrained_embedding_cfg.should_use=True agent.multitask.task_encoder_cfg.model_cfg.output_dim=6

System information

• MTRL Version : 1
• Python version : 3.6.13

Description

ModuleNotFoundError: No module named 'ml_logger.logbook'

System information

• ml_logger version : 0.7.17
• Python version : 3.6.13

Any other information

running CARE

Thank you for contributing to MTRL.

Proposed Change

Fixing typos related to HiP-BMDP

Have you read the Contributing Guidelines?

Yes

Related Issues/PRs

#4

Description

Hi, it's not an issue about running the code, but some possible typos in the implementation of HiP-BMDP. (so I just ignore the format...)

(1) About the configuration file ./config/agent/components/hipbmdp_multitask.yaml

The default setting is should_use_task_encoder: False. I wonder that, if we want to include environment encoding as a part of the input of the transition model (as Figure 2 in the paper), shall we set it to be True?

Besides, I tried to run the code after setting it to be true, to avoid dimension mismatch error when running the code, I also need to set should_condition_encoder_on_task_info and should_concatenate_task_info_with_encoder to be True in that file. I would like to know that if my current setting is correct for HiP-BMDP algorithm.

(2) maybe a typo in ./mtrl/agent/hipbmdp.py

In line 110, the second argument of function get_task_encoding() is mode, but when this function is called in ./mtrl/agent/sac.py Line 200, the second argument is specified as modes, and it will cause an error when running the code. I guess it's just a typo and I can run the code after fixing it.

Could you help to fix these issues, or please let me know if I misunderstand something? Thanks!

Description

I run the code as the doc described

PYTHONPATH=. python3 -u main.py \
setup=metaworld \
env=metaworld-mt10 \
agent=state_sac \
experiment.num_eval_episodes=1 \
experiment.num_train_steps=2000000 \
setup.seed=1 \
replay_buffer.batch_size=1280 \
agent.multitask.num_envs=10 \
agent.multitask.should_use_disentangled_alpha=True \
agent.encoder.type_to_select=identity \
agent.multitask.should_use_multi_head_policy=False \
agent.multitask.actor_cfg.should_condition_model_on_task_info=False \
agent.multitask.actor_cfg.should_condition_encoder_on_task_info=True \
agent.multitask.actor_cfg.should_concatenate_task_info_with_encoder=True

And it shows that

Multiple values for agent/optimizers. To override a value use 'override agent/optimizers: metaworld_encoder' Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

How can I solve this problem? Thanks~

How to reproduce

Add a minimal example to reproduce the issue.

Stack trace / error message

Paste the stack trace/error message to Gist and paste the link here.

System information

• MTRL Version :
• MTRL environment Name :
• Python version :

Any other information

Add any other information here.

Description

When can the mtrl repo support metaworld-v2, which has been released for an whole year and should be considered as the new standard environment for multi-task rl?

@shagunsodhani

How to reproduce

Add a minimal example to reproduce the issue.

Stack trace / error message

Paste the stack trace/error message to Gist and paste the link here.

System information

• MTRL Version :
• MTRL environment Name :
• Python version :

Any other information

Add any other information here.

Description

• setup: metaworld
• experiment: metaworld
• agent: sac
• agent/components: ${defaults.0.setup}_actor #- agent/components: ${defaults.0.setup}_critic #- agent/components: ${defaults.0.setup}_encoder #- agent/components: ${defaults.0.setup}_transition_model #- agent/components: task_to_encoder_cluster #- agent/components: ${defaults.0.setup}_mask #- agent/components: ${defaults.0.setup}_multitask #- agent/components: ${defaults.0.setup}_transition_model #- agent/components: ${defaults.0.setup}_gradnorm #- agent/components: ${defaults.0.setup}_task_to_encoder_cluster
• agent/optimizers: ${defaults.0.setup}_actor #- agent/optimizers: ${defaults.0.setup}_alpha #- agent/optimizers: ${defaults.0.setup}_critic #- agent/optimizers: ${defaults.0.setup}_decoder #- agent/optimizers: ${defaults.0.setup}_encoder
• env: metaworld-mt10
• replay_buffer: mtrl
• logger: mtrl
• metrics: ${defaults.0.setup}
• logbook: mtrl

Running log:

setup=metaworld agent=state_sac env=metaworld-mt10 agent.multitask.num_envs=10 agent.multitask.should_use_disentangled_alpha=True /home/zoulixin/miniconda3/lib/python3.9/site-packages/gym/envs/registration.py:440: UserWarning: WARN: The registry.env_specs property along with EnvSpecTree is deprecated. Please use registry directly as a dictionary instead. logger.warn( Traceback (most recent call last): File "/home/zoulixin/data/keras-mmoe/CAGrad-main/mtrl/temp/mtrl-main/main.py", line 7, in from mtrl.app.run import run File "/home/zoulixin/data/keras-mmoe/CAGrad-main/mtrl/temp/mtrl-main/mtrl/app/run.py", line 7, in from mtrl.experiment import utils as experiment_utils File "/home/zoulixin/data/keras-mmoe/CAGrad-main/mtrl/temp/mtrl-main/mtrl/experiment/utils.py", line 11, in from mtrl.experiment.metaworld import Experiment File "/home/zoulixin/data/keras-mmoe/CAGrad-main/mtrl/temp/mtrl-main/mtrl/experiment/metaworld.py", line 10, in from mtrl.env import builder as env_builder File "/home/zoulixin/data/keras-mmoe/CAGrad-main/mtrl/temp/mtrl-main/mtrl/env/builder.py", line 5, in import mtenv File "/home/zoulixin/miniconda3/lib/python3.9/site-packages/mtenv/init.py", line 5, in from mtenv.envs.registration import make # noqa: F401 File "/home/zoulixin/miniconda3/lib/python3.9/site-packages/mtenv/envs/init.py", line 10, in register( File "/home/zoulixin/miniconda3/lib/python3.9/site-packages/mtenv/envs/registration.py", line 74, in register return mtenv_registry.register(id, **kwargs) File "/home/zoulixin/miniconda3/lib/python3.9/site-packages/mtenv/envs/registration.py", line 66, in register self.env_specs[id] = MultitaskEnvSpec(id, **kwargs) File "/home/zoulixin/miniconda3/lib/python3.9/site-packages/mtenv/envs/registration.py", line 41, in init super().init( File "", line 12, in init AttributeError: can't set attribute

How to reproduce

Add a minimal example to reproduce the issue.

Stack trace / error message

Paste the stack trace/error message to Gist and paste the link here.

System information

• MTRL Version :
• MTRL environment Name :
• Python version :

Any other information

Add any other information here.

Description

I was following docs, and when I tried to use command

PYTHONPATH=. python3 -u main.py \
setup=metaworld \
agent=state_sac \
env=metaworld-mt10 \
agent.multitask.num_envs=10 \
agent.multitask.should_use_disentangled_alpha=True

I kept getting AttributeError: can't set attribute error.

How to reproduce

PYTHONPATH=. python3 -u main.py \
setup=metaworld \
agent=state_sac \
env=metaworld-mt10 \
agent.multitask.num_envs=10 \
agent.multitask.should_use_disentangled_alpha=True

and the full error message is here

System information

• MTRL Version : e282c3f5a205970b4ad7d1c1ebae7aa9b4d56218 commit
• MTRL environment Name : metaworld-mt10
• Python version : 3.7

Any other information

I use mujoco-py==2.0.2.0 and af8417bfc82a3e249b4b02156518d775f29eb289 commit for the MetaWorld environments, and I have also run through following two test codes to ensure my mujoco-py and metaworld was installed successfully. mujoco-py test:

import mujoco_py
import os
mj_path, _ = mujoco_py.utils.discover_mujoco()
xml_path = os.path.join(mj_path, 'model', 'humanoid.xml')
model = mujoco_py.load_model_from_path(xml_path)
sim = mujoco_py.MjSim(model)

print(sim.data.qpos)

metaworld test:

import metaworld
import random

mt10 = metaworld.MT10() # Construct the benchmark, sampling tasks

training_envs = []
for name, env_cls in mt10.train_classes.items():
  env = env_cls()
  task = random.choice([task for task in mt10.train_tasks
                        if task.env_name == name])
  env.set_task(task)
  training_envs.append(env)

for env in training_envs:
  obs = env.reset()  # Reset environment
  a = env.action_space.sample()  # Sample an action
  obs, reward, done, info = env.step(a)  # Step the environoment with the sampled random action

Also, the full step I create my environment is here

Description

An error was encountered after running the sample code for a while， I use such command line settings： PYTHONPATH=. python3 -u main.py
setup=metaworld
agent=state_sac
env=metaworld-mt10
agent.multitask.num_envs=10
agent.multitask.should_use_disentangled_alpha=True

How to reproduce

Error in call to target 'mtrl.experiment.metaworld.Experiment': ConfigAttributeError('Missing key encoder\n full_key: agent.encoder\n object_type=dict') full_key: experiment.builder

The above exception was the direct cause of the following exception:

File "C:\Users\Tz0942\Downloads\mtrl-main\mtrl\experiment\metaworld.py", line 20, in init super().init(config, experiment_id) File "C:\Users\Tz0942\Downloads\mtrl-main\mtrl\experiment\multitask.py", line 25, in init super().init(config, experiment_id) File "C:\Users\Tz0942\Downloads\mtrl-main\mtrl\experiment\experiment.py", line 63, in init self.config = prepare_config(config=self.config, env_metadata=self.env_metadata) File "C:\Users\Tz0942\Downloads\mtrl-main\mtrl\experiment\experiment.py", line 187, in prepare_config if key in config.agent.encoder:

The above exception was the direct cause of the following exception:

System information

• MTRL Version : e282c3f5a205970b4ad7d1c1ebae7aa9b4d56218
• MTRL environment Name : meatworld-mt10
• Python version : python 3.8
• metaworld commit: af8417bfc82a3e249b4b02156518d775f29eb289

Any other information

I have to modify some configuration items in config.yaml can be annotated before I run main.py successfully: config.yaml like this: defaults: - setup: metaworld - experiment: ${defaults.0.setup} - agent: sac - agent/components: ${defaults.0.setup}_actor #- agent/components: ${defaults.0.setup}_critic #- agent/components: ${defaults.0.setup}_encoder #- agent/components: ${defaults.0.setup}_transition_model #- agent/components: task_to_encoder_cluster #- agent/components: ${defaults.0.setup}_mask #- agent/components: ${defaults.0.setup}_multitask #- agent/components: ${defaults.0.setup}_transition_model #- agent/components: ${defaults.0.setup}_gradnorm #- agent/components: ${defaults.0.setup}_task_to_encoder_cluster - agent/optimizers: ${defaults.0.setup}_actor #- agent/optimizers: ${defaults.0.setup}_alpha #- agent/optimizers: ${defaults.0.setup}_critic #- agent/optimizers: ${defaults.0.setup}_decoder #- agent/optimizers: ${defaults.0.setup}_encoder - env: ??? - replay_buffer: mtrl - logger: mtrl - metrics: ${defaults.0.setup} - logbook: mtrl

Description

we have successfully trained some algorithm on MT10. However,when we train a sac agent on MT1 to get "One SAC agent per task(upper bound)", it always fails due to critic loss is to high( >1e8), and the success rate is near 0%. Is there any special config for MT1?

How to reproduce

we use the following config：

setup=metaworld
env=metaworld-mt1
agent=state_sac
experiment.num_eval_episodes=1
experiment.num_train_steps=2000000
setup.seed=10
replay_buffer.batch_size=1280
agent.multitask.num_envs=1
agent.multitask.should_use_disentangled_alpha=False
agent.encoder.type_to_select=identity
agent.multitask.should_use_multi_head_policy=False
agent.multitask.actor_cfg.should_condition_model_on_task_info=False
agent.multitask.actor_cfg.should_condition_encoder_on_task_info=True
agent.multitask.actor_cfg.should_concatenate_task_info_with_encoder=True

we change the default task_name in the function get_list_of_func_to_make_envs() ( src/mtenv/mtenv/envs/metaworld/env.py) to control the task uesd in MT1.

System information

• MTRL Version : latest
• Metaword Version : af8417bfc82a3e249b4b02156518d775f29eb289

.