Hello, I am currently interested in applying preference-based RL in various robotics tasks in the real world.

I read the paper "BPref" and noticed that the preference is given to the agent by various simulated teachers, not real humans, to make the training and evaluation process fast. But in your previous paper "PEBBLE", it seems that real human preferences were used.

I am curious what kind of GUI you used for the humans to interactively give preference feedback. Could you give me some tips?

cf) For the simulation environment, I am using RaiSim and Isaac gym.

I test the experiments on Intel Xeon Gold 5118 and GTX2080 Ti. But I found that the utilization rate of GPU is very low. It takes 10 hours to train the Walker_walk/train_ppo.sh for 3 seeds. Is it my experimental parameters or is it normal?

During the setup instructions, when creating the conda environment, the two dependencies for dm_control and hydra fail.

You can just manually install dm_control and that seems to work, but installing hydra manually seems to cause some errors downstream.

Error message during the setup:

(bpref) kuwajerw@pop-os [12:41:39PM 10/11/2022]:
(main) ~/repos/BPref/
$ pip install git+git://github.com/facebookresearch/[email protected]_branch
Collecting git+git://github.com/facebookresearch/[email protected]_branch
  Cloning git://github.com/facebookresearch/hydra (to revision 0.11_branch) to /tmp/pip-req-build-g15zht1f
  Running command git clone -q git://github.com/facebookresearch/hydra /tmp/pip-req-build-g15zht1f
  fatal: unable to connect to github.com:
  github.com[0: 140.82.113.4]: errno=Connection timed out

WARNING: Discarding git+git://github.com/facebookresearch/[email protected]_branch. Command errored out with exit status 128: git clone -q git://github.com/facebookresearch/hydra /tmp/pip-req-build-g15zht1f Check the logs for full command output.
ERROR: Command errored out with exit status 128: git clone -q git://github.com/facebookresearch/hydra /tmp/pip-req-build-g15zht1f Check the logs for full command output.
(bpref) kuwajerw@pop-os [12:43:55PM 10/11/2022]:
(main) ~/repos/BPref/
$ pip install git+git://github.com/deepmind/dm_control.git@d1fe1d14fb229d3ad784e40d8c54491d30e1f08a
Collecting git+git://github.com/deepmind/dm_control.git@d1fe1d14fb229d3ad784e40d8c54491d30e1f08a
  Cloning git://github.com/deepmind/dm_control.git (to revision d1fe1d14fb229d3ad784e40d8c54491d30e1f08a) to /tmp/pip-req-build-fe8j117j
  Running command git clone -q git://github.com/deepmind/dm_control.git /tmp/pip-req-build-fe8j117j
  fatal: unable to connect to github.com:
  github.com[0: 140.82.112.4]: errno=Connection timed out

WARNING: Discarding git+git://github.com/deepmind/dm_control.git@d1fe1d14fb229d3ad784e40d8c54491d30e1f08a. Command errored out with exit status 128: git clone -q git://github.com/deepmind/dm_control.git /tmp/pip-req-build-fe8j117j Check the logs for full command output.
ERROR: Command errored out with exit status 128: git clone -q git://github.com/deepmind/dm_control.git /tmp/pip-req-build-fe8j117j Check the logs for full command output.
(bpref) kuwajerw@pop-os [12:52:36PM 10/11/2022]:
(main) ~/repos/BPref/
$ 

Error message after manually installing hydra 0.11 (the version that this repo uses I believe)

So first if I comment out the hydra line during the installation and run the code I obviously get a "hydra not found error"

/home/kuwajerw/anaconda3/envs/bpref/lib/python3.6/site-packages/gym/logger.py:34: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize("%s: %s" % ("WARN", msg % args), "yellow"))
Traceback (most recent call last):
  File "train_SAC.py", line 13, in <module>
    import hydra
ModuleNotFoundError: No module named 'hydra'

Then I install hydra 0.11

(bpref) kuwajerw@pop-os [01:02:30PM 10/11/2022]:
(main) ~/repos/BPref/
$ pip install hydra-core==0.11
Collecting hydra-core==0.11
  Using cached hydra_core-0.11.0-py3-none-any.whl (71 kB)
Collecting omegaconf>=1.4.0
  Using cached omegaconf-2.2.3-py3-none-any.whl (79 kB)
Requirement already satisfied: PyYAML>=5.1.0 in /home/kuwajerw/anaconda3/envs/bpref/lib/python3.6/site-packages (from omegaconf>=1.4.0->hydra-core==0.11) (6.0)
Requirement already satisfied: dataclasses in /home/kuwajerw/anaconda3/envs/bpref/lib/python3.6/site-packages (from omegaconf>=1.4.0->hydra-core==0.11) (0.8)
Collecting antlr4-python3-runtime==4.9.*
  Using cached antlr4_python3_runtime-4.9.3-py3-none-any.whl
Installing collected packages: antlr4-python3-runtime, omegaconf, hydra-core
Successfully installed antlr4-python3-runtime-4.9.3 hydra-core-0.11.0 omegaconf-2.2.3

But then when I try to run one of the demo scripts, I get an error in hydra about not being able to import "Config"

(bpref) kuwajerw@pop-os [01:02:35PM 10/11/2022]:
(main) ~/repos/BPref/
$ ./scripts/button_press/run_sac.sh
/home/kuwajerw/anaconda3/envs/bpref/lib/python3.6/site-packages/gym/logger.py:34: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize("%s: %s" % ("WARN", msg % args), "yellow"))
Traceback (most recent call last):
  File "train_SAC.py", line 13, in <module>
    import hydra
  File "/home/kuwajerw/anaconda3/envs/bpref/lib/python3.6/site-packages/hydra/__init__.py", line 2, in <module>
    from . import utils
  File "/home/kuwajerw/anaconda3/envs/bpref/lib/python3.6/site-packages/hydra/utils.py", line 7, in <module>
    from hydra.plugins.common.utils import HydraConfig
  File "/home/kuwajerw/anaconda3/envs/bpref/lib/python3.6/site-packages/hydra/plugins/__init__.py", line 11, in <module>
    from .completion_plugin import CompletionPlugin
  File "/home/kuwajerw/anaconda3/envs/bpref/lib/python3.6/site-packages/hydra/plugins/completion_plugin.py", line 7, in <module>
    from omegaconf import DictConfig, ListConfig, Config, MissingMandatoryValue
ImportError: cannot import name 'Config'

Maybe we can use a different hydra version? That also seems to get other errors, I will update this issue.

Edit: So if I manually use hydra version 1.0, I get this error.

(bpref) kuwajerw@pop-os [01:20:11PM 10/11/2022]:
(main) ~/repos/BPref/
$ ./scripts/button_press/run_sac.sh
/home/kuwajerw/anaconda3/envs/bpref/lib/python3.6/site-packages/gym/logger.py:34: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize("%s: %s" % ("WARN", msg % args), "yellow"))

Key 'name' not in 'HydraConf'
    full_key: hydra.name
    object_type=HydraConf

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

Hi,

First of all, I want to congratulate you on the excellent work.

I was wondering if it would be possible to share also the code that was used to get the plots with rliable?

Hi, it seems that the version.txt is missing in /stable_baseline3, which will cause a error when installing (running pip install -e .[docs,tests,extra])