This PR aims for:

1. Create a Python script that calls run_rep_learner.py first then il_train.py. This file should allow the normal workflow of pretrain -> adapt, and also grid search over different configurations.

2. Modify relevant files. e.g. Saving and loading models

This is a draft PR sketching out the code to implement the tests in Hypothesis 3 of our experiment spreadsheet.

It attempts to implement:

For each benchmark: --> For each environment: -----> For each dataset structure:

if MAGICAL:

• train k seeds of RepL for each algo, then subsequently k*j seeds of IL on those pretrained RepL

if DMC:

• train k*j seeds of RepL+IL without reuse

Changes the repL data-loading pipeline to use a new, uniform on-disk format. This should eventually allow us to do multi-task training; to train on imitation policy rollouts; to train on random rollouts; etc.

Things that this PR does, or will do:

• Introduces a new demonstration format based on webdataset.
• Refactors the repl experiment to exclusively read demonstrations in this format.
• Creates mkdataset_demos.py script that can convert the "legacy" data formats into the new unified format.
• Creates mkdataset_random.py to generating random rollouts in the new unified format.
• Splits the benchmark Sacred ingredient into three parts: env, venv_opts, and env_data. The first contains all (and only) the options necessary to construct environments. The second contains additional options necessary to configure vec envs. The last contains hardcoded paths to data of different types.
• Adds config options to the repl experiment that allows it to use multiple types of data, rather than just single-task demos.

This PR does not make il_train.py use the new data format, although in principle that should be possible (just unnecessary, since we're not doing multi-task training).

Also, a warning: I had to rewrite the target pair constructors and parts of the repL training loop to work with the new dataset format. There might be some bugs related to those things which our tests have not caught. If reviewing this, pay special attention to those parts.

This PR makes GAIL work again, and makes a few related changes too:

• Updates imitation/SB3 deps (this means people will need to reinstall them from requirements.txt; should ping Slack when this is merged).
• Makes MineCraft tests skippable if minerl is not installed. This approach would be easy to extend to make dm_control skippable if MuJoCo is not installed, make MAGICAL skippable if no X server is running, etc.
• Solves MoveToCorner

There are also some unrelated things it does (because I wanted to use the fixes above without making a new PR):

• Slightly adjusts VAE hyperparams so that VAE actually works
• Adds some experiment configs that I was using to test GAIL
• Adds data generation scripts for ICML
• Fixes some issues with the skopt tuning code which needed to be fixed in order to tune GAIL

Things it doesn't do:

• Get DMC environments to reach adequate reward. This needs a bit more hyperparameter tuning.

This is a minimal change to allow augmentation to happen on the dataset rather than the batch level. This is probably going to be provisional (either due to changes in Sam's PR, or due to changes to use Kornia), so I expect this to change, but wanted to make a MVP version of this functionality exist earlier for CIFAR testing use.

This PR:

• Adds logic to only create a new repl seed if one hasn't already been set. This makes it possible to set the seed explicitly in the config (in which case it will be part of the encoder-lookup hash), or else leave it empty (in which case it will not be part of the hash, and will be randomly generated)
• More rigorously recursively sorts config dictionaries before jsonpickle-ing and hashing them
• Adds logic for an is_multitask flag, and ensures that if a RepL task is set as multitask, its env_cfg.task_name isn't included in its config hash

I have not yet confirmed the consistency of hashes across OSes, but plan to do so tomorrow morning, at which point I'll move this PR out of draft status, but since we're under time constraints, it seemed good to put what had been done up for review now.

[Note: I spent quite awhile trying to implement a test that would check the env_cfg multitask functionality, but ran into difficulties because Magical is the setting where we'd be planning to do Multitask, since DMC has different action sizes, and Magical still errors out on my machine due to the NSFWindow bug]

Copied from the docs:

Like InverseDynamicsPrediction, except it uses a contrastive loss function.

During the decoder stage, instead of predicting an action, we simply concatenate the representations of s and s' together. During the encoder stage, we need to also encode the actions into a vector of size 2 * representation_dim, so that they can be contrasted against the concatenation of encodings of s and s'.

Pointing imitation requirement to https://github.com/HumanCompatibleAI/imitation/pull/259 's branch for now. Will update requirements to new commit once https://github.com/HumanCompatibleAI/imitation/pull/259 is merged into imitation's ILR dev branch.

This PR:

• Refactors the decoder projection head to learn a standard deviation (when it does learn one) as a function of the encoder-predicted mean, not the encoder-predicted standard deviation. This avoids the "NN on top of an exponential" effect that we believe to have been the cause of exploding values
• Changes the stochastic parameter of the encoder to learn_scale, so that the decoder and encoder have consistent parameters, which both correspond to "should I learn a standard deviation". In the encoder case, not learning one results in returning a constant. In the decoder case, not learning one results in returning whatever standard deviation was returned by the encoder.

This PR has the goal of writing hyperparameter tuning scripts that can be run per-environment, to try to find optimal hyperparameters on a per-env, per-algorithm basis. This involved:

• Moving hyperparameter tuning logic to live in its own file, separate from pretrain_n_adapt
• Defining search spaces for some new algorithms
• Writing a bash script to run everything together

This PR contains a number of small changes implemented in the process of running CEB tests.

• Creates a MultiLogger object, that is a lightweight wrapper around both a logger and a Torch SummaryWriter, and which can be more easily passed into objects to allow them to add to logs. I've currently only added to augmenter and loss constructor
• Adds a TargetProjection decoder which only adds a projection layer to the target, and not to the context, to allow for symmetry-breaking between the two, and heuristic approximation of learnt bilinear loss
• Allows for reading in expert trajectories [Now deprecated, given Sam's changes to Master]
• Cleans up Sacred parameters such that it's easier to add ad-hoc parameters without needing to use --force (which had been necessary if you added a parameter to the algo hyperparameter dictionary)
• Moves some kwarg-handling logic from __init__ into RepresentationLearner

Hi @qxcv, first of all thanks a lot of open-sourcing the codebase for your amazing work. The codebase is indeed huge, I was wondering if this repository contains code for end-to-end Imitation Learning without any Representation Learning (i.e. w/o Pre-training/Join Training).

Also, do you have a small dataset so that I can check if it works on my end in small scale?

I also see that you have provided two datasets, can you please explain which involves which tasks?

Thanks!

This PR works to make Minecraft environments more generally compatible with the ILR training framework, including those environments that have Dict Action/Environment spaces

NOTE: This is a draft PR until the branches torch_conversion and then ilr_wrappers get merged into realistic_benchmarks, since the functionality here depends on implements new to those branches.

Modifications in this branch:

• Modify Minecraft data-loading and env-loading code so that you can pass in an arbitrary set of wrappers in an env_cfg config entry, and have those wrap both the observations/actions coming from the loaded dataset, and also the actual live environment (Previously, there was single hardcoded wrapper applied to all Minecraft envs)
• Add config options for using a SpaceFlattenedActorCriticPolicy (implemented in realistic_benchmarks) for environments that require it (i.e. environments with Dict-like Action spaces)
• Modify our test configs list such that we only add Minecraft to the list of test configs when it is available as a benchmark (required because the actual config entry itself now requires an import from realistic_benchmarks, which relies on minerl being installed)

This cleans up some of the existing tooling and adds some more. Specifically:

• Removes requirements.txt and adds all deps to setup.py instead.
• Adds run_rep_learner, il_train and il_test commands to entry_points so that we don't have to do python src/il_representations/scripts/{name}.py.
• Automatically runs flake8 and isort as part of the unit tests. This uses the pytest-isort and pytest-flake8 plugins.
• Adds a reformat.sh script that automatically cleans up imports with isort and autoflake, then reformats to be PEP8-compliant with yapf. This can handle most of the problems identified by pytest-isort and pytest-flake8. My hope is that from now on, we can just run reformat.sh before committing but otherwise not have to worry about formatting at all (except wrapping strings and comments).

This PR also reformats ALL the code to be compliant with the new flake8 style. I've rebased everything into two commits: one that makes the tooling changes, and one that makes the (mostly automated) formatting changes. This should make it easier to review.

One potentially controversial choice I've made here is the line length of 100. I noticed that Cody and Cynthia's code was mostly around ~120 columns. I personally prefer shorter line lengths because I can fit more code on my screen; e.g. <=83 columns means I can fit five files, <=105 columns means I can fit four, and <~130 means I can fit three. I chose 100 as a compromise that makes it easy to fit several files on the screen at the same time without forcing us to wrap too much. I'm open to discussions about that choice, as well as discussions about how we can tweak yapf to make files look nice :smile:

Another thing: this PR requires everyone to uninstall & reinstall MAGICAL (if they have it installed already). I made some changes that are necessary to work with the latest version of Pyglet, but apparently those won't automatically get installed by pip if you have an existing version :disappointed:

Adds run_cifar.py, which runs representation learning with a ResNet-18 on CIFAR-10, finetunes a linear layer on top of it, and evaluates the accuracy of the resulting classifier. Hyperparameters are set to mimic SimCLR: https://github.com/google-research/simclr/

The current implementation still depends on an incorrect loss function (to be fixed in #10 ) and augments the examples at a batch level instead of a dataset level (to be fixed in a different PR).