TLaunch: Launch Programs on Multiple Hosts

Overview

TLaunch: Launch Programs on Multiple Hosts

Introduction

Deepmind launchpad is a library that helps writing distributed program in a simple way. But currently it only supports (or has only open-sourced) launching programs on a single host, either multi-threaded or multi-processed. This library provides a way of launching existing launchpad programs on multiple nodes. Only some simple modification to your program is needed.

Installation

First, let's clone this repository. Then cd into the repository, and execute:

pip install -r requirements.txt
pip install .

Usage

1. Launch programs on multiple hosts without communication

from absl import app
import logging
from tlaunch import lp_ssh

class Worker:
  def __init__(self, worker_id):
    self.worker_id = worker_id

  def run(self):
    logging.info('Worker {}:{}'.format(self.worker_id, i))
    lp_ssh.stop()

def make_program():
  program = lp_ssh.Program('worker')
  worker_num = 2
  current_num = 0
  for host in ['host1','host2']:
    for i in range(worker_num):
      ssh_node  = lp_ssh.SSHNode(Worker, current_num).to_host(host)
      current_num += 1
      program.add_node(ssh_node, label=host+'_worker')
  lp_ssh.launch(program, terminal='ssh_tmux_session')
def main(_):
  make_program()

if __name__ == '__main__':
  app.run(main)

In this code, we place Worker on host1 and host2 via to_host() function. With lp_ssh.launch(), Each Worker will start to run on its corresponding hosts. Besides, examples/mnist/run.sh shows an example of how to train MNIST dataset on multiple hosts.

2. Launch programs on multiple hosts with communication

examples/commands/run_cmd.py gives an example of how to check GPU status of remote hosts. The information can be transferred via defining a TransmitNode.

3. Different data-transfer types

[Optional] Kubernetes Support

If you want to use TLaunch with Kubernetes:

  1. go installed on host machine to run kustomize.
  2. A running kubernetes cluster.
  3. Volcano scheduler installed to enable gang scheduling. This will tell you how to install volcano for your kubernetes cluster.

Install and run lp-operator on your kubernetes cluster

cd lp-operator
make deploy

Then lp-operator should be running in namespace lp-operator-system. You can use kubectl get all -n lp-operator-system to check the status of the running operator.

[Optional] Kubernetes Usage

Check here for more information.

Citing TLaunch

If you use TLaunch in your work, please cite us:

@article{tartrl2021tlaunch,
    title={TLaunch: Launch Programs on Multiple Hosts},
    author={Shiyu Huang, Sen Na, Shizhen Xu, Ting Chen, Jun Zhu},
    year={2021},
    howpublished={\url{https://github.com/TARTRL/TLaunch}},
}
You might also like...
Python directory buster, multiple threads, gobuster-like CLI, web server brute-forcer, URL replace pattern feature.

pybuster v1.1 pybuster is a tool that is used to brute-force URLs of web servers. Features Directory busting (URI) URL replace patterns (put PYBUSTER

Fast and easy way to rollout on multiple GitLab project file a particular content.

Volatile Fast and easy way to rollout on multiple GitLab project file a particular content. Why ? After looking for a tool to simply enforce a develop

A tool to brute force a gmail account. Use this tool to crack multiple accounts
A tool to brute force a gmail account. Use this tool to crack multiple accounts

A tool to brute force a gmail account. Use this tool to crack multiple accounts. This tool is developed to crack multiple accounts

A wordlist generator tool, that allows you to supply a set of words, giving you the possibility to craft multiple variations from the given words, creating a unique and ideal wordlist to use regarding a specific target.
A wordlist generator tool, that allows you to supply a set of words, giving you the possibility to craft multiple variations from the given words, creating a unique and ideal wordlist to use regarding a specific target.

A wordlist generator tool, that allows you to supply a set of words, giving you the possibility to craft multiple variations from the given words, creating a unique and ideal wordlist to use regarding a specific target.

Daemon to ban hosts that cause multiple authentication errors

__ _ _ ___ _ / _|__ _(_) |_ ) |__ __ _ _ _ | _/ _` | | |/ /| '_ \/ _` | ' \

A server hosts a FastAPI application and multiple clients can be connected to it via SocketIO.

FastAPI_and_SocketIO A server hosts a FastAPI application and multiple clients can be connected to it via SocketIO. Executing server.py sets up the se

This repo is a C++ version of yolov5_deepsort_tensorrt. Packing all C++ programs into .so files, using Python script to call C++ programs further.

yolov5_deepsort_tensorrt_cpp Introduction This repo is a C++ version of yolov5_deepsort_tensorrt. And packing all C++ programs into .so files, using P

A minimal Streamlit app showing how to launch and stop a FastAPI process on demand
A minimal Streamlit app showing how to launch and stop a FastAPI process on demand

Simple Streamlit + FastAPI Integration A minimal Streamlit app showing how to launch and stop a FastAPI process on demand. The FastAPI /run route simu

Python script to launch burp scans automatically

SimpleAutoBurp Python script that takes a config.json file as config and uses Burp Suite Pro to scan a list of websites.

A launcher to launch games from Riot Games under Linux

rito-launcher A launcher to launch games from Riot Games under Linux Requirements: Python 3, with the following pip plugins: 'configparser, pathlib, w

Launch a ready-to-code Wagtail Live development environment with a single click.
Launch a ready-to-code Wagtail Live development environment with a single click.

Wagtail Live Gitpod Launch a ready-to-code Wagtail Live development environment with a single click. Steps: Click the Open in Gitpod button. Relax: a

Launcher program to select which version of the Q-Sys software to launch.
Launcher program to select which version of the Q-Sys software to launch.

QSC-QSYS Launcher Launcher program to select which version of the Q-Sys software to launch. Instructions To use the application simply save the "Q-Sys

Migration Manager (MM) is a very small utility that can list source servers in a target account and apply mass launch template modifications.

Migration Manager Migration Manager (MM) is a very small utility that can list source servers in a target account and apply mass launch template modif

AppQuickLauncher is a tool that can quickly launch apps by clicking the app tray icon.
AppQuickLauncher is a tool that can quickly launch apps by clicking the app tray icon.

AppQuickLauncher AppQuickLauncher is a tool that can quickly launch apps by clicking the app tray icon. On Windows 7 or Windows 10, we can add a folde

Launch any Heroic-Games-Launcher game using bash scripts without having to open Heroic.

HeroicBashLauncher Ever wanted to launch your EGS games installed through Heroic Games Launcher directly from the terminal, Lutris or any other fronte

A simple and lightweight server that allows clients to connect and launch a shell remotely through a browser.

carrotsh A simple and lightweight server that allows clients to connect and launch a shell remotely through a browser. Uses xterm.js for the frontend

Pancakeswap Sniper Bot GUI Uniswap Matic 2022 (WINDOWS LINUX MAC) AUTO BUY TOKEN ON LAUNCH AFTER ADD LIQUIDITY
Pancakeswap Sniper Bot GUI Uniswap Matic 2022 (WINDOWS LINUX MAC) AUTO BUY TOKEN ON LAUNCH AFTER ADD LIQUIDITY

Pancakeswap Sniper Bot GUI Uniswap Matic 2022 (WINDOWS LINUX MAC) ⭐️ AUTO BUY TOKEN ON LAUNCH AFTER ADD LIQUIDITY ⭐️ ⭐️ First GUI SNIPER BOT for WINDO

A tool for RaceRoom Racing Experience which shows you launch data

R3E Launch Tool A tool for RaceRoom Racing Experience which shows you launch data. Usage Run the tool, change the Stop Speed to whatever you want, and

Owner
Tsinghua AI Research Team for Reinforcement Learning
Tsinghua AI Research Team for Reinforcement Learning (Creativity, Practicality and Optimist)
Tsinghua AI Research Team for Reinforcement Learning
Consolidating and extending hosts files from several well-curated sources. You can optionally pick extensions to block pornography, social media, and other categories.

Take Note! With the exception of issues and PRs regarding changes to hosts/data/StevenBlack/hosts, all other issues regarding the content of the produ

Steven Black 22.1k Jan 2, 2023
DependConfusion-X Tool is written in Python3 that scans and monitors list of hosts for Dependency Confusion

DependConfusion-X Tool is written in Python3 which allows security researcher/bug bounty hunter to scan and monitor list of hosts for Dependency Confusion.

Ali Fathi Ali Sawehli 4 Dec 21, 2021
Python library to remotely extract credentials on a set of hosts.

Python library to remotely extract credentials on a set of hosts.

Pixis 1.5k Dec 31, 2022
LittleBrother is a simple parental control application monitoring specific processes on Linux hosts to monitor and limit the play time of children.

Parental Control Application LittleBrother Overview LittleBrother is a simple parental control application monitoring specific processes (read "games"

null 40 Dec 21, 2022
Proof of concept to check if hosts are vulnerable to CVE-2021-41773

CVE-2021-41773 PoC Proof of concept to check if hosts are vulnerable to CVE-2021-41773. Description (https://cve.mitre.org/cgi-bin/cvename.cgi?name=CV

Jordan Jay 43 Nov 9, 2022
A simple automatic tool for finding vulnerable log4j hosts

Log4Scan A simple automatic tool for finding vulnerable log4j hosts Installation pip3 install -r requirements.txt Usage usage: log4scan.py [-h] (-f FI

Federico Rapetti 20018955 6 Mar 10, 2022
A fully automated, accurate, and extensive scanner for finding vulnerable log4j hosts

log4j-scan A fully automated, accurate, and extensive scanner for finding vulnerable log4j hosts Features Support for lists of URLs. Fuzzing for more

Duc Linh Nguyen 4 Aug 8, 2022
Coerce authentication from Windows hosts via MS-FSRVP (Requires FS-VSS-AGENT service running on host)

VSSTrigger Coerce authentication from Windows hosts via MS-FSRVP (Requires FS-VS

Filip Dragovic 6 Jul 24, 2022
A collection of over 5.1 million sub-domains and assets belonging to public bug bounty programs, compiled into a repo, for performing bulk operations.

?? Public Bug Bounty Targets Data By BugBountyResources A collection of over 5.1M sub-domains and assets belonging to bug bounty targets, all put in a

Bug Bounty Resources 87 Dec 13, 2022
A deobfuscator for multiple python obfuscators

PY4COC A deobfuscator for multiple python obfuscators, supports exe's packed with pyinstaller too. How to use python3 py4coc.py <exe file or py file o

svenskithesource 16 Dec 3, 2022