A job launching library for docker, EC2, GCP, etc.

Related tags

DevOps Tools doodad
Overview

doodad

License: GPL v3 Build Status codecov

A library for packaging dependencies and launching scripts (with a focus on python) on different platforms using Docker. Currently supported platforms include AWS, GCP, and remotely via SSH.

doodad is designed to be as minimally invasive as possible - most code can be run without any modifications.

See the quickstart guide for a quick tutorial.

Setup

  • Install Python 3.6+

  • Install Docker CE.

  • Add this repo to your pythonpath.

export PYTHONPATH=$PYTHONPATH:/path/to/this/repo
  • Install dependencies
pip install -r requirements.txt

Example

Launching commands is very simple with doodad. The following script will launch a docker container and execute the command echo helloworld:

from doodad.launch import launch_api

launch_api.run_command('echo helloworld')

To launch a python program, specify the path to the script.

from doodad.launch import launch_api

launch_api.run_python('path/to/my/python/script.py')

Misc

EC2 code is based on rllab's code.

Issues
  • How are files copied with localdocker?

    How are files copied with localdocker?

    LocalDocker should at least be copying files from the host target mount into the /mounts/target directory right?

    However, the volume syntax -v host_dir:container_dir does not copy files. So how are we copying the python files such that the target_full_path in make_python_command is available?

    For example, at what file/line in the launch script at the end here copied:

    python /mounts/target/scripts/run_experiment_from_doodad.py

    opened by richardrl 1
  • Fix SSH functionality

    Fix SSH functionality

    (also temporarily disable EC2 as it is not fully implemented)

    opened by justinjfu 1
  • Sync Interval for GCP Mounts

    Sync Interval for GCP Mounts

    It seems like the snyc interval is not use for gcp mounts when building the archive. In the gsutil startup script, the sync is on a fixed 15 second interval which may be problematic for large files. Is there some way to use the sync interval argument in this setting?

    opened by btickell 1
  • Support skip wait on `call_and_wait`

    Support skip wait on `call_and_wait`

    This lets you launch a bunch of things from doodad in local mode asynchronously. Use at your own risk!

    opened by vitchyr 1
  • add singularity image support

    add singularity image support

    Currently it's defaulting to docker, unless singularity_image is specified.

    (my IDE also automatically removed some white spaces)

    opened by vitchyr 1
  • Multiple docker instances in one EC2 instance

    Multiple docker instances in one EC2 instance

    This lets you run multiple experiments on an EC2 instance. While this doesn't help for CPU instance, this helps a lot with GPU instances, where the GPU utilization may not be that high.

    opened by vitchyr 1
  • AWS fixes

    AWS fixes

    • Fixes script args not working with AWS
    • Adds AWS to the hyperparam sweeper
    opened by justinjfu 0
  • Remove trailing slash gcp

    Remove trailing slash gcp

    Idk why there's so many commits... but this fixes some annoying small bugs/typos with the GCP and singularity modes.

    opened by vitchyr 0
  • Move data_sync_interval to an argument

    Move data_sync_interval to an argument

    Allow users to adjust the data sync interval. This should help with large files which take a long time to sync.

    opened by justinjfu 0
  • Update launch_api.py

    Update launch_api.py

    null

    opened by Riderbeck 1
  • EC2 Mode permanently disabled

    EC2 Mode permanently disabled

    Seems like ec2 support was disabled in January and never re-enabled. Is this feature currently broken?

    opened by btickell 0
  • Dar archive takes a long time to run

    Dar archive takes a long time to run

    Hi, I'm using the input mode with MountLocal, and it seems that doodad get stuck on running compile_archive function.

    The data folder I'm using is around 86Mb, and doodad was stuck for >7mins and hasn't finished.

    Would this be expected behavior?

    Thanks!

    opened by hzyjerry 1
  • Missing functionality

    Missing functionality

    It seems get_args() no longer exists in the most recent commit. Is there a way to get arguments from sweeper? Thanks!

    opened by btickell 0
  • Refactoring

    Refactoring

    opened by richardrl 0
  • small fix to launch_shell for LocalDocker mode

    small fix to launch_shell for LocalDocker mode

    mount_points was not being passed to LocalDocker.launch_command(), resulting in an error.

    opened by kylehkhsu 0
  • Fix to allow GPU allocations, ports & more arguments to launch_shell

    Fix to allow GPU allocations, ports & more arguments to launch_shell

    There are also some changes to the region specifications (i.e., there is no longer a difference between s3_bucket_region and region). Is this the intent, or should they be treated separately?

    opened by eringrant 0
Owner
Justin Fu
Justin Fu
A tool to convert AWS EC2 instances back and forth between On-Demand and Spot billing models.

ec2-spot-converter This tool converts existing AWS EC2 instances back and forth between On-Demand and 'persistent' Spot billing models while preservin

jcjorel 120 Oct 27, 2021
Python job scheduling for humans.

schedule Python job scheduling for humans. Run Python functions (or any other callable) periodically using a friendly syntax. A simple to use API for

Dan Bader 9.2k Dec 4, 2021
A Python library for the Docker Engine API

Docker SDK for Python A Python library for the Docker Engine API. It lets you do anything the docker command does, but from within Python apps – run c

Docker 5.5k Nov 24, 2021
Visual disk-usage analyser for docker images

whaler What? A command-line tool for visually investigating the disk usage of docker images Why? Large images are slow to move and expensive to store.

Treebeard Technologies 193 Nov 18, 2021
Define and run multi-container applications with Docker

Docker Compose Docker Compose is a tool for running multi-container applications on Docker defined using the Compose file format. A Compose file is us

Docker 24.3k Dec 2, 2021
Docker Container wallstreetbets-sentiment-analysis

Docker Container wallstreetbets-sentiment-analysis A docker container using restful endpoints exposed on port 5000 "/analyze" to gather sentiment anal

null 134 Nov 24, 2021
Hackergame nc 类题目的 Docker 容器资源限制、动态 flag、网页终端

Hackergame nc 类题目的 Docker 容器资源限制、动态 flag、网页终端 快速入门 配置证书 证书用于验证用户 Token。请确保这里的证书文件(cert.pem)与 Hackergame 平台 配置的证书相同,这样 Hackergame 平台为每个用户生成的 Token 才可以通

USTC Hackergame 58 Nov 25, 2021
docker-compose工程部署时的辅助脚本

okta-cmd Introduction docker-compose 辅助脚本

完美风暴666 4 Nov 25, 2021
🐳 Docker templates for various languages.

Docker Deployment Templates One Stop repository for Docker Compose and Docker Templates for Deployment. Features Python (FastAPI, Flask) Screenshots D

CodeChef-VIT 5 Oct 21, 2021
Push Container Image To Docker Registry In Python

push-container-image-to-docker-registry 概要 push-container-image-to-docker-registry は、エッジコンピューティング環境において、特定のエッジ端末上の Private Docker Registry に特定のコンテナイメー

Latona, Inc. 3 Nov 4, 2021
Lima is an alternative to using Docker Desktop on your Mac.

lima-xbar-plugin Table of Contents Description Installation Dependencies Lima is an alternative to using Docker Desktop on your Mac. Description This

Joe Block 18 Nov 9, 2021
Build Netbox as a Docker container

netbox-docker The Github repository houses the components needed to build Netbox as a Docker container. Images are built using this code and are relea

Farshad Nick 1 Oct 31, 2021
Some automation scripts to setup a deployable development database server (with docker).

Postgres-Docker Database Initializer This is a simple automation script that will create a Docker Postgres database with a custom username, password,

Pysogge 1 Nov 11, 2021
A repository containing a short tutorial for Docker (with Python).

Docker Tutorial for IFT 6758 Lab In this repository, we examine the advtanges of virtualization, what Docker is and how we can deploy simple programs

Arka Mukherjee 1 Dec 1, 2021
The leading native Python SSHv2 protocol library.

Paramiko Paramiko: Python SSH module Copyright: Copyright (c) 2009 Robey Pointer <[email protected]> Copyright: Copyright (c) 2020 Jeff Forcier <

null 7.3k Dec 3, 2021
Official Python client library for kubernetes

Kubernetes Python Client Python client for the kubernetes API. Installation From source: git clone --recursive https://github.com/kubernetes-client/py

Kubernetes Clients 4.3k Dec 2, 2021
Asynchronous parallel SSH client library.

parallel-ssh Asynchronous parallel SSH client library. Run SSH commands over many - hundreds/hundreds of thousands - number of servers asynchronously

null 953 Dec 3, 2021
CI repo for building Skia as a shared library

Automated Skia builds This repo is dedicated to building Skia binaries for use in Skija. Prebuilt binaries Prebuilt binaries can be found in releases.

Humble UI 8 Nov 21, 2021
Deploy an inference API on AWS (EC2) using FastAPI Docker and Github Actions

Deploy an inference API on AWS (EC2) using FastAPI Docker and Github Actions To learn more about this project: medium blog post The goal of this proje

Ahmed BESBES 32 Nov 22, 2021
Project repository of Apache Airflow, deployed on Docker in Amazon EC2 via GitLab.

Airflow on Docker in EC2 + GitLab's CI/CD Personal project for simple data pipeline using Airflow. Airflow will be installed inside Docker container,

Ammar Chalifah 7 Nov 20, 2021
Linux, Jenkins, AWS, SRE, Prometheus, Docker, Python, Ansible, Git, Kubernetes, Terraform, OpenStack, SQL, NoSQL, Azure, GCP, DNS, Elastic, Network, Virtualization. DevOps Interview Questions

Linux, Jenkins, AWS, SRE, Prometheus, Docker, Python, Ansible, Git, Kubernetes, Terraform, OpenStack, SQL, NoSQL, Azure, GCP, DNS, Elastic, Network, Virtualization. DevOps Interview Questions

Arie Bregman 19.5k Nov 27, 2021
Python process launching

sh is a full-fledged subprocess replacement for Python 2.6 - 3.8, PyPy and PyPy3 that allows you to call any program as if it were a function: from sh

Andrew Moffat 5.9k Dec 3, 2021
DirBruter is a Python based CLI tool. It looks for hidden or existing directories/files using brute force method. It basically works by launching a dictionary based attack against a webserver and analyse its response.

DirBruter DirBruter is a Python based CLI tool. It looks for hidden or existing directories/files using brute force method. It basically works by laun

vijay sahu 9 Sep 28, 2021
A framework for launching new Django Rest Framework projects quickly.

DRFx A framework for launching new Django Rest Framework projects quickly. Comes with a custom user model, login/logout/signup, social authentication

William Vincent 337 Nov 17, 2021
A tool to convert AWS EC2 instances back and forth between On-Demand and Spot billing models.

ec2-spot-converter This tool converts existing AWS EC2 instances back and forth between On-Demand and 'persistent' Spot billing models while preservin

jcjorel 120 Oct 27, 2021
Python codes for the server and client end that facilitates file transfers. (Using AWS EC2 instance as the server)

Server-and-Client-File-Transfer Python codes for the server and client end that facilitates file transfers. I will be using an AWS EC2 instance as the

Amal Farhad Shaji 2 Oct 13, 2021