A job launching library for docker, EC2, GCP, etc.

Related tags

DevOps Tools doodad
Overview

doodad

License: GPL v3 Build Status codecov

A library for packaging dependencies and launching scripts (with a focus on python) on different platforms using Docker. Currently supported platforms include AWS, GCP, and remotely via SSH.

doodad is designed to be as minimally invasive as possible - most code can be run without any modifications.

See the quickstart guide for a quick tutorial.

Setup

  • Install Python 3.6+

  • Install Docker CE.

  • Add this repo to your pythonpath.

export PYTHONPATH=$PYTHONPATH:/path/to/this/repo
  • Install dependencies
pip install -r requirements.txt

Example

Launching commands is very simple with doodad. The following script will launch a docker container and execute the command echo helloworld:

from doodad.launch import launch_api

launch_api.run_command('echo helloworld')

To launch a python program, specify the path to the script.

from doodad.launch import launch_api

launch_api.run_python('path/to/my/python/script.py')

Misc

EC2 code is based on rllab's code.

Comments
  • Sync Interval for GCP Mounts

    Sync Interval for GCP Mounts

    It seems like the snyc interval is not use for gcp mounts when building the archive. In the gsutil startup script, the sync is on a fixed 15 second interval which may be problematic for large files. Is there some way to use the sync interval argument in this setting?

    opened by btickell 1
  • How are files copied with localdocker?

    How are files copied with localdocker?

    LocalDocker should at least be copying files from the host target mount into the /mounts/target directory right?

    However, the volume syntax -v host_dir:container_dir does not copy files. So how are we copying the python files such that the target_full_path in make_python_command is available?

    For example, at what file/line in the launch script at the end here copied:

    python /mounts/target/scripts/run_experiment_from_doodad.py

    opened by richardrl 1
  • Multiple docker instances in one EC2 instance

    Multiple docker instances in one EC2 instance

    This lets you run multiple experiments on an EC2 instance. While this doesn't help for CPU instance, this helps a lot with GPU instances, where the GPU utilization may not be that high.

    opened by vitchyr 1
  • Google Cloud Platform

    Google Cloud Platform

    Will require going through these first to setup GCP:

    https://cloud.google.com/sdk/docs/quickstart-debian-ubuntu https://cloud.google.com/compute/docs/tutorials/python-guide https://cloud.google.com/storage/docs/reference/libraries#client-libraries-install-python

    opened by stevenlin1111 0
  • Generate script to launch slurm experiments

    Generate script to launch slurm experiments

    This way you can:

    1. generate a script locally
    2. scp it over
    3. launch the script from the cluster.

    The reason you might want to do this is so that you don't have to (e.g.) install MuJoCo and other dependencies on the cluster. You just launch the script and the DOODAD_ARGS already has the dependencies serialized (which will be deserialized by the Singularity image running the experiment, and not by the interactive node).

    opened by vitchyr 0
  • Support other regions

    Support other regions

    The script right now assumes that the EC2 instance and the S3 bucket are in the same region. This current design forces you to create a new bucket every time you want to launch an EC2 instance on another region.

    This commit lets you specific the region of EC2 and s3 separately.

    opened by vitchyr 0
  • Add option to override image id and aws_s3_path

    Add option to override image id and aws_s3_path

    This doesn't change default behavior. It just exposes more options so that you can (e.g.) make an auto-config docker but change the the image id and/or aws s3 path.

    opened by vitchyr 0
  • Fix ec2 setup & few small changes

    Fix ec2 setup & few small changes

    Looks like pycharm made a bunch of formatting updates as well (sorry!). Here are the main fixes/changes:

    • scripts/setup_ec2.py Fix typo in setup_ec2 script
    • doodad/launch_tools.py: Fix bug in local mode that caused a directory to be mounted to a directory that might not exist. (docker doesn't have this issue since it calls (mkdir -p) on that directory)
    • doodad/arg_parse.py: default to use cloudpickle to allow more flexibility in what args can be passed to script
    • doodad/mode.py:
      • can set the "experiment name" directory, so that instead of saving things to an s3 folder called my_exp/run-123123123, it's called my-exp/my-exp-123123123123123 (by default) or my-exp/my-custom-exp-name-123123123 (if you set experiment name). This will also make it so that the ec2 instance name is my-custom-exp-name-123123123.
      • remove references to log_dir since this is a vestige of rllab.
      • Sync everything to s3 on termination. If things were working before, I think it's either because you happened to name your mount point /tmp/expt or because your experiment was long enough so that a periodic sync would save your results before the script terminated. Honestly not 100% sure how things worked before, but I think this code is much more clear.
    opened by vitchyr 0
  • Dar archive takes a long time to run

    Dar archive takes a long time to run

    Hi, I'm using the input mode with MountLocal, and it seems that doodad get stuck on running compile_archive function.

    The data folder I'm using is around 86Mb, and doodad was stuck for >7mins and hasn't finished.

    Would this be expected behavior?

    Thanks!

    opened by hzyjerry 1
  • Fix to allow GPU allocations, ports & more arguments to launch_shell

    Fix to allow GPU allocations, ports & more arguments to launch_shell

    There are also some changes to the region specifications (i.e., there is no longer a difference between s3_bucket_region and region). Is this the intent, or should they be treated separately?

    opened by eringrant 0
Owner
Justin Fu
Justin Fu
A tool to convert AWS EC2 instances back and forth between On-Demand and Spot billing models.

ec2-spot-converter This tool converts existing AWS EC2 instances back and forth between On-Demand and 'persistent' Spot billing models while preservin

jcjorel 152 Dec 29, 2022
Build and Push docker image in Python (luigi + docker-py)

Docker build images workflow in Python Since docker hub stopped building images for free accounts, I've been looking for another way to do it. I could

Fabien D. 2 Dec 15, 2022
Python IMDB Docker - A docker tutorial to containerize a python script.

Python_IMDB_Docker A docker tutorial to containerize a python script. Build the docker in the current directory: docker build -t python-imdb . Run the

Sarthak Babbar 1 Dec 30, 2021
Python job scheduling for humans.

schedule Python job scheduling for humans. Run Python functions (or any other callable) periodically using a friendly syntax. A simple to use API for

Dan Bader 10.4k Jan 2, 2023
A Python library for the Docker Engine API

Docker SDK for Python A Python library for the Docker Engine API. It lets you do anything the docker command does, but from within Python apps – run c

Docker 6.1k Dec 31, 2022
Visual disk-usage analyser for docker images

whaler What? A command-line tool for visually investigating the disk usage of docker images Why? Large images are slow to move and expensive to store.

Treebeard Technologies 194 Sep 1, 2022
Define and run multi-container applications with Docker

Docker Compose Docker Compose is a tool for running multi-container applications on Docker defined using the Compose file format. A Compose file is us

Docker 28.2k Jan 8, 2023
Docker Container wallstreetbets-sentiment-analysis

Docker Container wallstreetbets-sentiment-analysis A docker container using restful endpoints exposed on port 5000 "/analyze" to gather sentiment anal

null 145 Nov 22, 2022
Hackergame nc 类题目的 Docker 容器资源限制、动态 flag、网页终端

Hackergame nc 类题目的 Docker 容器资源限制、动态 flag、网页终端 快速入门 配置证书 证书用于验证用户 Token。请确保这里的证书文件(cert.pem)与 Hackergame 平台 配置的证书相同,这样 Hackergame 平台为每个用户生成的 Token 才可以通

USTC Hackergame 68 Nov 9, 2022
docker-compose工程部署时的辅助脚本

okta-cmd Introduction docker-compose 辅助脚本

完美风暴666 4 Dec 9, 2021
🐳 Docker templates for various languages.

Docker Deployment Templates One Stop repository for Docker Compose and Docker Templates for Deployment. Features Python (FastAPI, Flask) Screenshots D

CodeChef-VIT 6 Aug 28, 2022
Push Container Image To Docker Registry In Python

push-container-image-to-docker-registry 概要 push-container-image-to-docker-registry は、エッジコンピューティング環境において、特定のエッジ端末上の Private Docker Registry に特定のコンテナイメー

Latona, Inc. 3 Nov 4, 2021
Lima is an alternative to using Docker Desktop on your Mac.

lima-xbar-plugin Table of Contents Description Installation Dependencies Lima is an alternative to using Docker Desktop on your Mac. Description This

Joe Block 68 Dec 22, 2022
Build Netbox as a Docker container

netbox-docker The Github repository houses the components needed to build Netbox as a Docker container. Images are built using this code and are relea

Farshad Nick 1 Dec 18, 2021
Some automation scripts to setup a deployable development database server (with docker).

Postgres-Docker Database Initializer This is a simple automation script that will create a Docker Postgres database with a custom username, password,

Pysogge 1 Nov 11, 2021
A repository containing a short tutorial for Docker (with Python).

Docker Tutorial for IFT 6758 Lab In this repository, we examine the advtanges of virtualization, what Docker is and how we can deploy simple programs

Arka Mukherjee 0 Dec 14, 2021
Bitnami Docker Image for Python using snapshots for the system packages repositories

Python Snapshot packaged by Bitnami What is Python Snapshot? Python is a programming language that lets you work quickly and integrate systems more ef

Bitnami 1 Jan 13, 2022
Hatch plugin for Docker containers

hatch-containers CI/CD Package Meta This provides a plugin for Hatch that allows

Ofek Lev 11 Dec 30, 2022
Create pinned requirements.txt inside a Docker image using pip-tools

Pin your Python dependencies! pin-requirements.py is a script that lets you pin your Python dependencies inside a Docker container. Pinning your depen

null 4 Aug 18, 2022