Nb workflows - A workflow platform which allows you to run parameterized notebooks programmatically

Overview

NB Workflows

readthedocs PyPI - Format PyPI - Status

Description

If SQL is a lingua franca for querying data, Jupyter should be a lingua franca for data explorations, model training, and complex and unique tasks related to data.

This workflow platform allows to run parameterized notebooks programmatically. Notebooks could be scheduled with cron syntax, by intervals, run by n times, only once or in time ranges (from 10am to 18pm).

So, the notebook is the main UI and the workflow job description.

Goal

Empowering different data roles in a project to put code into production, simplifying the time required to do so. It enables people to go from a data exploration instance to an entirely pipeline deployed in production, using the same notebook file made by a data scientist, analyst or whatever role working with data in an iterative way.

How to run

# launch web process
# swagger by default: http://localhost:8000/docs/swagger 
make web 
# RQ Worker
make rqworker
# RQScheduler (optional)
make rqscheduler

Architecture

nb_workflows architecture

References & inspirations

Comments
  • Support for Python 3.10

    Support for Python 3.10

    I'm using Python 3.10, but labfunctions is only supported on 3.8 and 3.9

    python -V
    Python 3.10.4
    
    pip install labfunctions[client]
    ERROR: Could not find a version that satisfies the requirement labfunctions[client] (from versions: none)
    ERROR: No matching distribution found for labfunctions[client]
    
    opened by munshkr 0
  • Runtimes

    Runtimes

    • Generation of Dockerfiles with cli
    • Difference between gpu runtimes and cpu runtimes
    • Registration of each runtime
    • Option to run notebooks in different runtimes
    opened by nuxion 0
  • Decouple redis instance from ClusterControl and cluster cli simplific…

    Decouple redis instance from ClusterControl and cluster cli simplific…

    …ation

    • Redis is only used by AgentRegister, in the future AgentRegister could be an interface and be used for HTTP and Redis commns
    • Cluster cli simplification given an unified interface: cluster is an optional parameter instead of an argument
    • default_cluster param in ClusterSpec File
    • option to use a CLUSTER_SPEC environment to avoit repeating -f argument in cluster cli
    opened by nuxion 0
  • CICD fixing tests

    CICD fixing tests

    It fix some issues related to libcloud, dependencies and environments vars when testing inside github actions.

    Also some scripts for quick-prerelealising added.

    opened by nuxion 0
  • release 0.7.0

    release 0.7.0

    Changelog

    • [x] Executios: Namespacing Execution ID for tracking loads.
    • [x] Executors: Execution ID injected from Web
    • [x] Notifications: slack and discord keep working on this for the next release
    • [x] Tests: >= 40%
    • [x] History: Execution result of a Workflow
    • [x] Web: API versioning
    • [x] Models: Alembic migrations implemented
    • [x] Copy outputs executions locally
    • [x] Project: During first time project creation review feedback if project name already exist
    • [x] Add project as mixin to History Model which allows to see full executions detail per project
    • [x] Log execution streaming
    • [x] Custom Errors
    • [x] Example project using nb-workflows
    • [x] Split NBClient into UserClient and AgentClient
    • [x] Tracks dockerfiles versions.
    • [x] Types: NBTask as pydantic model.
    • [x] Types: ScheduleData as pydantic model.
    • [x] Projects: File spec change where each Workflow Name will be a dict (like serivces in docker-compose.yml)
    • [x] Clients refactoring: One client for command line (with filesystem side effects), Another one as agent.

    Details The goal of this release is a functional system that delivery the promise of remote execution of notebook for production loads. With that in mind, the focus will be in the stabilization of workflows executions, adding tests cases, execution feedback and cli enhancements.

    opened by nuxion 0
  • Cannot build local image if user is not logged in

    Cannot build local image if user is not logged in

    Following the quickstart, I had an unknown error when building an image, I suspect this is because I never logged in:

    => Not private key found for this project
    => Bundling runtime default
    => Bundle generated in /home/munshkr/.labfunctions/pzj2xqb80g/.nb_tmp/default.current.zip
    (x) Error uploading file
    

    When I tried to build it locally, it threw an exception regarding no access token:

    => Not private key found for this project
    => Bundling runtime default
    => Bundle generated in /home/munshkr/.labfunctions/pzj2xqb80g/.nb_tmp/default.current.zip
    Traceback (most recent call last):
      File "/home/munshkr/dym/lab-test/.venv/bin/lab", line 8, in <module>
        sys.exit(cli())
      File "/home/munshkr/dym/lab-test/.venv/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
        return self.main(*args, **kwargs)
      File "/home/munshkr/dym/lab-test/.venv/lib/python3.10/site-packages/click/core.py", line 1055, in main
        rv = self.invoke(ctx)
      File "/home/munshkr/dym/lab-test/.venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/home/munshkr/dym/lab-test/.venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/home/munshkr/dym/lab-test/.venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/home/munshkr/dym/lab-test/.venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
        return __callback(*args, **kwargs)
      File "/home/munshkr/dym/lab-test/.venv/lib/python3.10/site-packages/labfunctions/cmd/runtimes.py", line 165, in build
        os.environ["LF_AGENT_TOKEN"] = c.creds.access_token
      File "/home/munshkr/dym/lab-test/.venv/lib/python3.10/site-packages/labfunctions/client/base.py", line 205, in creds
        access_token=self._auth.access_token,
    AttributeError: 'NoneType' object has no attribute 'access_token'```
    opened by munshkr 0
Owner
Xavier Petit
Programmer, Devops, passionate for opensource and software in general.
Xavier Petit
BisQue is a web-based platform designed to provide researchers with organizational and quantitative analysis tools for 5D image data. Users can extend BisQue by implementing containerized ML workflows.

Overview BisQue is a web-based platform specifically designed to provide researchers with organizational and quantitative analysis tools for up to 5D

Vision Research Lab @ UCSB 26 Nov 29, 2022
This program writes christmas wish programmatically. It is using turtle as a pen pointer draw christmas trees and stars.

Introduction This is a simple program is written in python and turtle library. The objective of this program is to wish merry Christmas programmatical

Gunarakulan Gunaretnam 1 Dec 25, 2021
Orthogonal Over-Parameterized Training

The inductive bias of a neural network is largely determined by the architecture and the training algorithm. To achieve good generalization, how to effectively train a neural network is of great importance. We propose a novel orthogonal over-parameterized training (OPT) framework that can provably minimize the hyperspherical energy which characterizes the diversity of neurons on a hypersphere. See our previous work -- MHE for an in-depth introduction.

Weiyang Liu 11 Apr 18, 2022
Parameterized Explainer for Graph Neural Network

PGExplainer This is a Tensorflow implementation of the paper: Parameterized Explainer for Graph Neural Network https://arxiv.org/abs/2011.04573 NeurIP

Dongsheng Luo 89 Dec 12, 2022
Code for our CVPR 2021 Paper "Rethinking Style Transfer: From Pixels to Parameterized Brushstrokes".

Rethinking Style Transfer: From Pixels to Parameterized Brushstrokes (CVPR 2021) Project page | Paper | Colab | Colab for Drawing App Rethinking Style

CompVis Heidelberg 153 Jan 4, 2023
Implementation of parameterized soft-exponential activation function.

Soft-Exponential-Activation-Function: Implementation of parameterized soft-exponential activation function. In this implementation, the parameters are

Shuvrajeet Das 1 Feb 23, 2022
Request execution of Galaxy SARS-CoV-2 variation analysis workflows on input data you provide.

SARS-CoV-2 processing requests Request execution of Galaxy SARS-CoV-2 variation analysis workflows on input data you provide. Prerequisites This autom

useGalaxy.eu 17 Aug 13, 2022
High performance Cross-platform Inference-engine, you could run Anakin on x86-cpu,arm, nv-gpu, amd-gpu,bitmain and cambricon devices.

Anakin2.0 Welcome to the Anakin GitHub. Anakin is a cross-platform, high-performance inference engine, which is originally developed by Baidu engineer

null 514 Dec 28, 2022
naked is a Python tool which allows you to strip a model and only keep what matters for making predictions.

naked is a Python tool which allows you to strip a model and only keep what matters for making predictions. The result is a pure Python function with no third-party dependencies that you can simply copy/paste wherever you wish.

Max Halford 24 Dec 20, 2022
Teaching end to end workflow of deep learning

Deep-Education This repository is now available for public use for teaching end to end workflow of deep learning. This implies that learners/researche

Data Lab at College of William and Mary 2 Sep 26, 2022
Alfred-Restore-Iterm-Arrangement - An Alfred workflow to restore iTerm2 window Arrangements

Alfred-Restore-Iterm-Arrangement This alfred workflow will list avaliable iTerm2

null 7 May 10, 2022
The pyrelational package offers a flexible workflow to enable active learning with as little change to the models and datasets as possible

pyrelational is a python active learning library developed by Relation Therapeutics for rapidly implementing active learning pipelines from data management, model development (and Bayesian approximation), to creating novel active learning strategies.

Relation Therapeutics 95 Dec 27, 2022
Fast and customizable reconnaissance workflow tool based on simple YAML based DSL.

Fast and customizable reconnaissance workflow tool based on simple YAML based DSL, with support of notifications and distributed workload of that work

Américo Júnior 3 Mar 11, 2022
This is an easy python software which allows to sort images with faces by gender and after by age.

Gender-age Classifier This is an easy python software which allows to sort images with faces by gender and after by age. Usage First install Deepface

Claudio Ciccarone 6 Sep 17, 2022
tinykernel - A minimal Python kernel so you can run Python in your Python

tinykernel - A minimal Python kernel so you can run Python in your Python

fast.ai 37 Dec 2, 2022
Spam your friends and famly and when you do your famly will disown you and you will have no friends.

SpamBot9000 Spam your friends and family and when you do your family will disown you and you will have no friends. Terms of Use Disclaimer: Please onl

DJ15 0 Jun 9, 2022
Theano is a Python library that allows you to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently. It can use GPUs and perform efficient symbolic differentiation.

============================================================================================================ `MILA will stop developing Theano <https:

null 9.6k Dec 31, 2022
a delightful machine learning tool that allows you to train, test and use models without writing code

igel A delightful machine learning tool that allows you to train/fit, test and use models without writing code Note I'm also working on a GUI desktop

Nidhal Baccouri 3k Jan 5, 2023