It seems like the snyc interval is not use for gcp mounts when building the archive. In the gsutil startup script, the sync is on a fixed 15 second interval which may be problematic for large files. Is there some way to use the sync interval argument in this setting?

LocalDocker should at least be copying files from the host target mount into the /mounts/target directory right?

However, the volume syntax -v host_dir:container_dir does not copy files. So how are we copying the python files such that the target_full_path in make_python_command is available?

For example, at what file/line in the launch script at the end here copied:

python /mounts/target/scripts/run_experiment_from_doodad.py

This lets you run multiple experiments on an EC2 instance. While this doesn't help for CPU instance, this helps a lot with GPU instances, where the GPU utilization may not be that high.

Will require going through these first to setup GCP:

https://cloud.google.com/sdk/docs/quickstart-debian-ubuntu https://cloud.google.com/compute/docs/tutorials/python-guide https://cloud.google.com/storage/docs/reference/libraries#client-libraries-install-python

This way you can:

1. generate a script locally
2. scp it over
3. launch the script from the cluster.

The reason you might want to do this is so that you don't have to (e.g.) install MuJoCo and other dependencies on the cluster. You just launch the script and the DOODAD_ARGS already has the dependencies serialized (which will be deserialized by the Singularity image running the experiment, and not by the interactive node).

The script right now assumes that the EC2 instance and the S3 bucket are in the same region. This current design forces you to create a new bucket every time you want to launch an EC2 instance on another region.

This commit lets you specific the region of EC2 and s3 separately.

This doesn't change default behavior. It just exposes more options so that you can (e.g.) make an auto-config docker but change the the image id and/or aws s3 path.

Looks like pycharm made a bunch of formatting updates as well (sorry!). Here are the main fixes/changes:

• scripts/setup_ec2.py Fix typo in setup_ec2 script
• doodad/launch_tools.py: Fix bug in local mode that caused a directory to be mounted to a directory that might not exist. (docker doesn't have this issue since it calls (mkdir -p) on that directory)
• doodad/arg_parse.py: default to use cloudpickle to allow more flexibility in what args can be passed to script
• doodad/mode.py:
  • can set the "experiment name" directory, so that instead of saving things to an s3 folder called my_exp/run-123123123, it's called my-exp/my-exp-123123123123123 (by default) or my-exp/my-custom-exp-name-123123123 (if you set experiment name). This will also make it so that the ec2 instance name is my-custom-exp-name-123123123.
  • remove references to log_dir since this is a vestige of rllab.
  • Sync everything to s3 on termination. If things were working before, I think it's either because you happened to name your mount point /tmp/expt or because your experiment was long enough so that a periodic sync would save your results before the script terminated. Honestly not 100% sure how things worked before, but I think this code is much more clear.

Hi, I'm using the input mode with MountLocal, and it seems that doodad get stuck on running compile_archive function.

The data folder I'm using is around 86Mb, and doodad was stuck for >7mins and hasn't finished.

Would this be expected behavior?

Thanks!

There are also some changes to the region specifications (i.e., there is no longer a difference between s3_bucket_region and region). Is this the intent, or should they be treated separately?