PipeLayer is a lightweight Python pipeline framework

Related tags

PipeLayer
Overview

PipeLayer

PipeLayer is a lightweight Python pipeline framework. Define a series of steps, and chain them together to create modular applications.

Table of Contents

Installation

From the command line:

pip install pipelayer

Getting Started

Step 1: Create Pipeline Filters

hello_world_filters.py

from pipelayer import Filter


class HelloFilter(Filter):
    def run(self, data, context):
        return "Hello"


class WorldFilter(Filter):
    def run(self, data, context):
        return f"{data},  World!"

functions.py

def create_message_dict(data, context):
    return {"message": data}

Step 2: Create a Pipeline

Create a module to run the pipeline:

app.py

from pipelayer import Pipeline

from functions import create_message
from hello_world_filters import HelloFilter, WorldFilter


if __name__ = "__main__":
    hello_world_pipeline = Pipeline([
        HelloFilter,                           # pipeline.Filter type
        WorldFilter,                           # pipeline.Filter instance
        create_message_dict                    # function type
        lambda data, context: json.dumps(data) # anonymous function
    ])

    output = hello_world_pipeline.run()

    # output = '{"message": "Hello, World!"}'

    print(f"Pipeline Output: {output}")
    print(hello_world_pipeline.manifest.__dict__)

Step 3: Run the Pipeline

from the command line:

run app.py

The Framework

pipelayer.Pipeline

__init__(steps, name)

args:

  • steps: List[Union[Step, Callable[[Any, Context], Any]]]
    A list of:

    • Classes and Instances that derive from pipelayer.Filter and implement the run method

    • Classes that implement the pipelayer.Step protocol

    • Functions (instance/class/static/module) that have the following signature

      def func(data: Any, context: Any)
    • Anonymous functions (lambda) with two arguments that follow this pattern:

      my_func = lambda data, context: data
    • Instances of pipelayer.Pipeline

  • name: Optional[str]
    If not specified, the class name will be used.

Properties:

name: str

state: Pipeline.State

steps: List[Union[Step, Callable[[Any, Context], Any]]]

manifest: Manifest
An instance of pipelayer.Manifest that is created when the run method is called.

Methods:

run(data, context) -> Any
The pipeline runner that iterates through the steps and pipes filter output to the next step.

args:

  • data: Any
  • context: pipelayer.Context

pipelayer.Switch

__init__(expression, cases, name)
An implementation of a Switch statement as a pipeline filter

args:

  • expression: Union[Step, Callable[[Any, Context], Any]]
  • cases: Dict[Union[Step, Callable[[Any, Context], Any]]]
  • name: Optional[str]
    If not specified, the class name will be used.

Properties:

expression: Union[Step, Callable[[Any, Context], Any]]
cases: Dict[Union[Step, Callable[[Any, Context], Any]]]
name: Optional[str]
manifest: Manifest

Methods:

run(data, context) -> Any
The switch runner that evaluates the specified expresssion executes the matching case.

pipelayer.Filter

__init__(name, pre_process, post_process)

args:

  • name: Optional[str]
    If not specified, the class name will be used.
  • pre_process: Optional[Callable[[Any, Context], Any]
  • post_process: Optional[Callable[[Any, Context], Any]

Properties:

pre_process: Optional[Callable[[Any, Context], Any]
post_process: Optional[Callable[[Any, Context], Any]

Events:
Events are lists of callables assignable after instantiation and are raised if the pipelayer.filter.raise_events decorator is applied to the implementation of the run method.

start: List[Callable[[Filter, Any], Any]]
Raised before the run method is invoked.

exit: List[Callable[[Filter, Any], Any]]
Raised if action is set to Action.SKIP or Action.EXIT in either a start or stop event handler.

end: List[Callable[[Filter, Any], Any]]
Raised after the run method is invoked.

pipelayer.FilterEventArgs

__init__(data, context, state)

args:

  • data: Any
  • context: Context
  • state: State

pipelayer.Context

A abstract base class for runtime app data.

pipelayer.Manifest

The Manifest keeps a record of Pipeline and Filter activity.

Utilities

pipelayer.util.render_manifest(manifest, indent) -> str
Static function that renders formatted JSON data

args:

  • manifest: Manifest
  • indent: Optional[int]
    Default value is 2.
Issues
  • Python 3.7 - unable to run sample code

    Python 3.7 - unable to run sample code

    Hi,

    With Python 3.7 I cannot run sample code. I have below error:

    Traceback (most recent call last): File "C:/Github/Python/pipelayers/app.py", line 1, in from pipelayer import Pipeline File "C:\Github\Python\venv\lib\site-packages\pipelayer_init_.py", line 6, in from pipelayer.pipeline import Pipeline # NOQA F401 File "C:\Github\Python\venv\lib\site-packages\pipelayer\pipeline.py", line 10, in from pipelayer.protocol import IFilter, IStep File "C:\Github\Python\venv\lib\site-packages\pipelayer\protocol.py", line 5, in from typing import (Any, Callable, List, Optional, Protocol, Tuple, ImportError: cannot import name 'Protocol' from 'typing' (C:\Program Files\Python\lib\typing.py)

    opened by zizu1985 3
  • Version 0.4.1

    Version 0.4.1

    • Minor bug squashed checking for a static run method.
    opened by greater-than 0
  • step_end event

    step_end event

    null

    opened by aebenson 0
  • Version 0.6.0

    Version 0.6.0

    opened by aebenson 0
  • Update Workflow

    Update Workflow

    opened by greater-than 0
  • Adds Python 3.7 to workflow

    Adds Python 3.7 to workflow

    opened by aebenson 0
  • Create python-app.yml

    Create python-app.yml

    opened by aebenson 0
  • Version 0.5.1

    Version 0.5.1

    opened by greater-than 0
  • Version 0.5.0

    Version 0.5.0

    • Adds Filter events/actions
    • Adds StepProxy for filters not implementing IStep
    opened by greater-than 0
Releases(v0.7.0)
Owner
greaterthan
greaterthan
The no-nonsense, minimalist REST and app backend framework for Python developers, with a focus on reliability, correctness, and performance at scale.

The Falcon Web Framework Falcon is a reliable, high-performance Python web framework for building large-scale app backends and microservices. It encou

Falconry 8.6k Oct 21, 2021
Lemon is an async and lightweight API framework for python

Lemon is an async and lightweight API framework for python . Inspired by Koa and Sanic .

Joway 28 Jul 26, 2021
Django Ninja - Fast Django REST Framework

Django Ninja is a web framework for building APIs with Django and Python 3.6+ type hints.

Vitaliy Kucheryaviy 1.6k Oct 23, 2021
The little ASGI framework that shines. ?

✨ The little ASGI framework that shines. ✨ Documentation: https://www.starlette.io/ Community: https://discuss.encode.io/c/starlette Starlette Starlet

Encode 6.2k Oct 24, 2021
Appier is an object-oriented Python web framework built for super fast app development.

Joyful Python Web App development Appier is an object-oriented Python web framework built for super fast app development. It's as lightweight as possi

Hive Solutions 116 Oct 2, 2021
bottle.py is a fast and simple micro-framework for python web-applications.

Bottle: Python Web Framework Bottle is a fast, simple and lightweight WSGI micro web-framework for Python. It is distributed as a single file module a

Bottle Micro Web Framework 7.4k Oct 23, 2021
Sierra is a lightweight Python framework for building and integrating web applications

A lightweight Python framework for building and Integrating Web Applications. Sierra is a Python3 library for building and integrating web applications with HTML and CSS using simple enough syntax. You can develop your web applications with Python, taking advantage of its functionalities and integrating them to the fullest.

null 74 Oct 8, 2021
Endpoints is a lightweight REST api framework written in python and used in multiple production systems that handle millions of requests daily.

Endpoints Quickest API builder in the West! Endpoints is a lightweight REST api framework written in python and used in multiple production systems th

Jay Marcyes 30 Oct 12, 2021
APIFlask is a lightweight Python web API framework based on Flask and marshmallow-code projects

APIFlask APIFlask is a lightweight Python web API framework based on Flask and marshmallow-code projects. It's easy to use, highly customizable, ORM/O

Grey Li 316 Oct 15, 2021
The web framework for inventors

Emmett is a full-stack Python web framework designed with simplicity in mind. The aim of Emmett is to be clearly understandable, easy to be learned an

Emmett 716 Oct 17, 2021
A familiar HTTP Service Framework for Python.

Responder: a familiar HTTP Service Framework for Python Powered by Starlette. That async declaration is optional. View documentation. This gets you a

Taoufik 3.6k Oct 15, 2021
Fast⚡, simple and light💡weight ASGI micro🔬 web🌏-framework for Python🐍.

NanoASGI Asynchronous Python Web Framework NanoASGI is a fast ⚡ , simple and light ?? weight ASGI micro ?? web ?? -framework for Python ?? . It is dis

Kavindu Santhusa 5 Oct 3, 2021
Pretty tornado wrapper for making lightweight REST API services

CleanAPI Pretty tornado wrapper for making lightweight REST API services Installation: pip install cleanapi Example: Project folders structure: . ├──

Vladimir Kirievskiy 17 Oct 4, 2021
The Modern And Developer Centric Python Web Framework. Be sure to read the documentation and join the Slack channel questions: http://slack.masoniteproject.com

NOTE: Masonite 2.3 is no longer compatible with the masonite-cli tool. Please uninstall that by running pip uninstall masonite-cli. If you do not unin

Masonite 1.7k Oct 23, 2021
The Python micro framework for building web applications.

Flask Flask is a lightweight WSGI web application framework. It is designed to make getting started quick and easy, with the ability to scale up to co

The Pallets Projects 56.9k Oct 17, 2021
WebSocket and WAMP in Python for Twisted and asyncio

Autobahn|Python WebSocket & WAMP for Python on Twisted and asyncio. Quick Links: Source Code - Documentation - WebSocket Examples - WAMP Examples Comm

Crossbar.io 2.3k Oct 12, 2021
FastAPI framework, high performance, easy to learn, fast to code, ready for production

FastAPI framework, high performance, easy to learn, fast to code, ready for production Documentation: https://fastapi.tiangolo.com Source Code: https:

Sebastián Ramírez 37.2k Oct 23, 2021
Async Python 3.6+ web server/framework | Build fast. Run fast.

Sanic | Build fast. Run fast. Build Docs Package Support Stats Sanic is a Python 3.6+ web server and web framework that's written to go fast. It allow

Sanic Community Organization 15.5k Oct 21, 2021
Async Python 3.6+ web server/framework | Build fast. Run fast.

Sanic | Build fast. Run fast. Build Docs Package Support Stats Sanic is a Python 3.6+ web server and web framework that's written to go fast. It allow

Sanic Community Organization 15.5k Oct 21, 2021