The easiest way to automate your data

Overview

Hello, world! 👋

We've rebuilt data engineering for the data science era.

Prefect is a new workflow management system, designed for modern infrastructure and powered by the open-source Prefect Core workflow engine. Users organize Tasks into Flows, and Prefect takes care of the rest.

Read the docs; get the code; ask us anything!

Welcome to Workflows

Prefect's Pythonic API should feel familiar for newcomers. Mark functions as tasks and call them on each other to build up a flow.

from prefect import task, Flow, Parameter


@task(log_stdout=True)
def say_hello(name):
    print("Hello, {}!".format(name))


with Flow("My First Flow") as flow:
    name = Parameter('name')
    say_hello(name)


flow.run(name='world') # "Hello, world!"
flow.run(name='Marvin') # "Hello, Marvin!"

For more detail, please see the Core docs

UI and Server

In addition to the Prefect Cloud platform, Prefect includes an open-source backend for orchestrating and managing flows, consisting primarily of Prefect Server and Prefect UI. This local server stores flow metadata in a Postgres database and exposes a GraphQL API.

Before running the server for the first time, run prefect backend server to configure Prefect for local orchestration. Please note the server requires Docker and Docker Compose to be running.

To start the server, UI, and all required infrastructure, run:

prefect server start

Once all components are running, you can view the UI by visiting http://localhost:8080.

Please note that executing flows from the server requires at least one Prefect Agent to be running: prefect agent local start.

Finally, to register any flow with the server, call flow.register(). For more detail, please see the orchestration docs.

"...Prefect?"

From the Latin praefectus, meaning "one who is in charge", a prefect is an official who oversees a domain and makes sure that the rules are followed. Similarly, Prefect is responsible for making sure that workflows execute properly.

It also happens to be the name of a roving researcher for that wholly remarkable book, The Hitchhiker's Guide to the Galaxy.

Integrations

Thanks to Prefect's growing task library and deep ecosystem integrations, building data applications is easier than ever.

Something missing? Open a feature request or contribute a PR! Prefect was designed to make adding new functionality extremely easy, whether you build on top of the open-source package or maintain an internal task library for your team.

Task Library

Airtable

Asana

AWS

Azure

Azure ML

Databricks

DBT

Docker

Dremio

Dropbox

Email

GitHub

Google Cloud

Google Sheets

Great Expectations

Jira

Jupyter

Kubernetes

Monday

MySQL

PostgreSQL

Python

Pushbullet

Redis

RSS

Shell

Slack

Snowflake

SpaCy

SQLite

SQL Server

Trello

Twitter

Deployment & Execution

Azure

AWS

Dask

Docker

Google Cloud

Kubernetes

Universal Deploy

Resources

Prefect provides a variety of resources to help guide you to a successful outcome.

We are committed to ensuring a positive environment, and all interactions are governed by our Code of Conduct.

Documentation

Prefect's documentation -- including concepts, tutorials, and a full API reference -- is always available at docs.prefect.io.

Instructions for contributing to documentation can be found in the development guide.

Slack Community

Join our Slack to chat about Prefect, ask questions, and share tips.

Blog

Visit the Prefect Blog for updates and insights from the Prefect team.

Support

Prefect offers a variety of community and premium support options for users of both Prefect Core and Prefect Cloud.

Contributing

Read about Prefect's community or dive in to the development guides for information about contributions, documentation, code style, and testing.

Installation

Requirements

Prefect requires Python 3.6+. If you're new to Python, we recommend installing the Anaconda distribution.

Latest Release

To install Prefect, run:

pip install prefect

or, if you prefer to use conda:

conda install -c conda-forge prefect

or pipenv:

pipenv install --pre prefect

Bleeding Edge

For development or just to try out the latest features, you may want to install Prefect directly from source.

Please note that the master branch of Prefect is not guaranteed to be compatible with Prefect Cloud or the local server.

git clone https://github.com/PrefectHQ/prefect.git
pip install ./prefect

License

Prefect Core is licensed under the Apache Software License Version 2.0. Please note that Prefect Core includes utilities for running Prefect Server and the Prefect UI, which are themselves licensed under the Prefect Community License.

Issues
  • Enter the Prefect 5k contest!

    Enter the Prefect 5k contest!

    Thanks for helping Prefect hit a big milestone!

    To celebrate 5,000 stars, we're having a contest.

    The grand prize is a $5,000 Cloud license, but we've got plenty of other items: Prefect gear, Marvin ducks, coffee, etc.

    It's really easy to enter: just mention @marvin-robot in any issue or PR on this repo. Even this one - go ahead and try!

    You can also enter by mentioning @Marvin in our Slack.

    Good luck and happy engineering!

    opened by jlowin 53
  • Use 'from' to explicitly chain exceptions

    Use 'from' to explicitly chain exceptions

    Current behavior

    I wanted to spend a little time contributing this weekend, so I ran pylint over this repo. I see this project has a .pylintrc but it looks like pylint isn't run in CI, so some things do show up.

    I found one class of warning from pylint that I think is worth addressing: explicitly re-raising exceptions in try-catches with from.

    The newest version of pylint added a check for this. You can see details in "Explicit Exception Chaining". That PEP describes a way to improve stack traces from try-catched exceptions.

    without from

    try:
        {}["a"]
    except KeyError:
        raise RuntimeError("hey this does not exist")
    
    Traceback (most recent call last):
      File "<stdin>", line 2, in <module>
    KeyError: 'a'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "<stdin>", line 4, in <module>
    RuntimeError: hey this does not exist
    

    with from

    try:
        {}["a"]
    except KeyError  as err:
        raise RuntimeError("hey this does not exist") from err
    
    Traceback (most recent call last):
      File "<stdin>", line 2, in <module>
    KeyError: 'a'
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "<stdin>", line 4, in <module>
    RuntimeError: hey this does not exist
    

    I think this language of "was the direct cause of the following exception" is clearer, and helps you to understand exactly where things started going wrong.

    Proposed behavior

    Would you be open to a pull request that changes try-catched exceptions in prefect to use this pattern? I wanted to ask first because right now there are 1334 places where that happens so it would touch a lot of code.

    you can see them all like this:

    pylint src/prefect/ | grep raise-missing-from > pylint.txt
    

    I think this change would improve prefect and help save users some debugging time. Especially users who are not very experienced in Python.

    Thanks for your time and consideration.

    Notes for Hacktoberfest Contributors

    This issue has been left open for Hacktoberfest 2020 contributors. If that describes you, welcome!

    1. Leave a comment below claiming a sub-module
    2. Identify all cases of this issue in that submodule by running code like the following in a terminal:
      pylint src/prefect/engine/cloud | grep "raise-missing-from"
      
    3. Fix all of these cases.
      • see #3320 for an example
    4. Open a pull request, following the steps in https://github.com/PrefectHQ/prefect/pulls
      • if there is already a changes/issue3306.yaml, add your name to the list of contributors
      • if there is not already a changes/issue3306.yaml, create one in your pull request, like in #3320

    sub-modules with this problem

    • ~src/prefect/client~ - @jameslamb (#3320)
    • ~src/prefect/core~ - @brainless (#3383)
    • ~src/prefect/engine~ - @brainless (#3383)
    • src/prefect/environments/execution/dask
    • src/prefect/environments/execution/k8s
    • ~src/prefect/environments/storage~ - @shalika10
    • src/prefect/tasks/airtable
    • ~src/prefect/tasks/aws~ - @heyitskevin
    • src/prefect/tasks/azure
    • src/prefect/tasks/azureml
    • ~src/prefect/tasks/databricks~ - @rikturr (#3448)
    • src/prefect/tasks/dbt
    • src/prefect/tasks/dropbox
    • ~src/prefect/tasks/gcp~ - @ericlundy87 (#3326)
    • src/prefect/tasks/great_expectations
    • src/prefect/tasks/gsheets
    • src/prefect/tasks/kubernetes
    • ~src/prefect/tasks/mysql~ - @luthrap (#3428)
    • ~src/prefect/tasks/postgres~ - @brunocasarotti
    • src/prefect/tasks/redis
    • src/prefect/tasks/rss
    • src/prefect/tasks/snowflake
    • ~src/prefect/tasks/spacy~ - @sp1thas
    • src/prefect/tasks/template
    • src/prefect/tasks/twitter
    • ~src/prefect/utilities~ - @gabrielcalderon (#3429)

    NOTE: This issue is not urgent. It's ok to claim a sub-module now and not contribute until Hacktoberfest begins on October 1st.

    good first issue 
    opened by jameslamb 43
  • Could not find a version that matches marshmallow-oneofschema<3.0,>=2.0.0b2

    Could not find a version that matches marshmallow-oneofschema<3.0,>=2.0.0b2

    $ python --version
    Python 3.5.2
    
    (prefect) [email protected]:~/code/prefect$ pipenv install prefect
    Installing prefect…
    Collecting prefect
      Downloading https://files.pythonhosted.org/packages/c3/86/6599cd54083875df820e3f980a22788a3e23756d8878f3b0e31a61b74dfe/prefect-0.5.0-py3-none-any.whl (208kB)
    Collecting marshmallow-oneofschema<3.0,>=2.0.0b2 (from prefect)
      Downloading https://files.pythonhosted.org/packages/d6/e4/9e95f3aeb376dd1e6ca65c72985e26d65e0124fe4d2d9623ad7d28fc1bc8/marshmallow_oneofschema-2.0.0b2-py2.py3-none-any.whl
    Collecting python-dateutil<3.0,>2.7.3 (from prefect)
      Downloading https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl (226kB)
    Collecting pendulum<3.0,>=2.0.4 (from prefect)
      Downloading https://files.pythonhosted.org/packages/95/37/91ea39046ea2523d72e0607a94e3ededd65f0d945ccb0e042f0aecd87dfc/pendulum-2.0.4-cp35-cp35m-manylinux1_x86_64.whl (139kB)
    Collecting distributed<2.0,>=1.21.8 (from prefect)
      Downloading https://files.pythonhosted.org/packages/e7/52/08ead8f0968d42510630f307100307b481298696cb7de81d58f0cdf7f320/distributed-1.26.0-py2.py3-none-any.whl (506kB)
    Collecting requests<3.0,>=2.20 (from prefect)
      Downloading https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl (57kB)
    Collecting xxhash<2.0,>=1.2.0 (from prefect)
      Downloading https://files.pythonhosted.org/packages/5e/bc/dc4c2f71a85b2952097fcf9c45475c7919cfe94290928a3467070dbec495/xxhash-1.3.0-cp35-cp35m-manylinux1_x86_64.whl (46kB)
    Collecting toml<1.0,>=0.9.4 (from prefect)
      Downloading https://files.pythonhosted.org/packages/a2/12/ced7105d2de62fa7c8fb5fce92cc4ce66b57c95fb875e9318dba7f8c5db0/toml-0.10.0-py2.py3-none-any.whl
    Collecting docker<4.0,>=3.4.1 (from prefect)
      Downloading https://files.pythonhosted.org/packages/fb/d8/8242b8fb3bd3000274fbf5ac1a06cdba8a5ccbcf4e2a8c05f0ab37999fd8/docker-3.7.1-py2.py3-none-any.whl (134kB)
    Collecting cryptography<3.0,>=2.2.2 (from prefect)
      Downloading https://files.pythonhosted.org/packages/5b/12/b0409a94dad366d98a8eee2a77678c7a73aafd8c0e4b835abea634ea3896/cryptography-2.6.1-cp34-abi3-manylinux1_x86_64.whl (2.3MB)
    Collecting pytz>=2018.7 (from prefect)
      Downloading https://files.pythonhosted.org/packages/61/28/1d3920e4d1d50b19bc5d24398a7cd85cc7b9a75a490570d5a30c57622d34/pytz-2018.9-py2.py3-none-any.whl (510kB)
    Collecting mypy-extensions<0.5,>=0.4.0 (from prefect)
      Downloading https://files.pythonhosted.org/packages/4d/72/8d54e2b296631b9b14961d583e56e90d9d7fba8a240d5ce7f1113cc5e887/mypy_extensions-0.4.1-py2.py3-none-any.whl
    Collecting cloudpickle<0.7,>=0.6.0 (from prefect)
      Downloading https://files.pythonhosted.org/packages/fc/87/7b7ef3038b4783911e3fdecb5c566e3a817ce3e890e164fc174c088edb1e/cloudpickle-0.6.1-py2.py3-none-any.whl
    Collecting idna<2.8,>=2.5 (from prefect)
      Downloading https://files.pythonhosted.org/packages/4b/2a/0276479a4b3caeb8a8c1af2f8e4355746a97fab05a372e4a2c6a6b876165/idna-2.7-py2.py3-none-any.whl (58kB)
    Collecting marshmallow==3.0.0b19 (from prefect)
      Downloading https://files.pythonhosted.org/packages/60/00/4f792fdbb7f0f243ce7fdb729bee3a8afde968e4bfda8365d47a9367a787/marshmallow-3.0.0b19-py2.py3-none-any.whl (48kB)
    Collecting mypy<0.700,>=0.600 (from prefect)
      Downloading https://files.pythonhosted.org/packages/88/11/8092fdd9cf4c507e6c799bf663e713a5418beb9fda422df810f72641224c/mypy-0.670-py3-none-any.whl (1.5MB)
    Collecting dask<2.0,>=0.18 (from prefect)
      Downloading https://files.pythonhosted.org/packages/b9/bc/0d747625c18397ed548c7890bf984a40d931b9ebac236c570a07565b0cc8/dask-1.1.4-py2.py3-none-any.whl (704kB)
    Collecting croniter<0.4,>=0.3 (from prefect)
      Downloading https://files.pythonhosted.org/packages/a9/c9/11182a2507798c661b04a7914739ea8ca73a738e6869a23742029f51bc1a/croniter-0.3.29-py2.py3-none-any.whl
    Collecting typing<4.0,>=3.6.4 (from prefect)
      Downloading https://files.pythonhosted.org/packages/4a/bd/eee1157fc2d8514970b345d69cb9975dcd1e42cd7e61146ed841f6e68309/typing-3.6.6-py3-none-any.whl
    Collecting typing-extensions<4.0,>=3.6.4 (from prefect)
      Downloading https://files.pythonhosted.org/packages/0f/62/c66e553258c37c33f9939abb2dd8d2481803d860ff68e635466f12aa7efa/typing_extensions-3.7.2-py3-none-any.whl
    Collecting click<8.0,>=7.0 (from prefect)
      Downloading https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl (81kB)
    Collecting pyyaml<4.3,>=3.13 (from prefect)
      Downloading https://files.pythonhosted.org/packages/9e/a3/1d13970c3f36777c583f136c136f804d70f500168edc1edea6daa7200769/PyYAML-3.13.tar.gz (270kB)
    Collecting six>=1.5 (from python-dateutil<3.0,>2.7.3->prefect)
      Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
    Collecting pytzdata>=2018.3 (from pendulum<3.0,>=2.0.4->prefect)
      Downloading https://files.pythonhosted.org/packages/89/02/a3a1cef5074c28157df63846d05aa893f007a92f6bafec0d61cae36bf69d/pytzdata-2018.9-py2.py3-none-any.whl (981kB)
    Collecting toolz>=0.7.4 (from distributed<2.0,>=1.21.8->prefect)
      Downloading https://files.pythonhosted.org/packages/14/d0/a73c15bbeda3d2e7b381a36afb0d9cd770a9f4adc5d1532691013ba881db/toolz-0.9.0.tar.gz (45kB)
    Collecting zict>=0.1.3 (from distributed<2.0,>=1.21.8->prefect)
      Downloading https://files.pythonhosted.org/packages/bd/45/a2e6f95a850cd407d785f2f8624b02e72baf6ab910aea4bdcac7e18b4871/zict-0.1.4-py2.py3-none-any.whl
    Collecting sortedcontainers!=2.0.0,!=2.0.1 (from distributed<2.0,>=1.21.8->prefect)
      Downloading https://files.pythonhosted.org/packages/13/f3/cf85f7c3a2dbd1a515d51e1f1676d971abe41bba6f4ab5443240d9a78e5b/sortedcontainers-2.1.0-py2.py3-none-any.whl
    Collecting tornado>=5 (from distributed<2.0,>=1.21.8->prefect)
      Downloading https://files.pythonhosted.org/packages/03/3f/5f89d99fca3c0100c8cede4f53f660b126d39e0d6a1e943e95cc3ed386fb/tornado-6.0.2.tar.gz (481kB)
    Collecting tblib (from distributed<2.0,>=1.21.8->prefect)
      Downloading https://files.pythonhosted.org/packages/4a/82/1b9fba6e93629a8557f9784cd8f1ae063c8762c26446367a6764edd328ce/tblib-1.3.2-py2.py3-none-any.whl
    Collecting msgpack (from distributed<2.0,>=1.21.8->prefect)
      Downloading https://files.pythonhosted.org/packages/3d/44/4a8be4f56ab3c8fc58800c06b2d681d5622704746d094411f00e25300072/msgpack-0.6.1-cp35-cp35m-manylinux1_x86_64.whl (243kB)
    Collecting psutil>=5.0 (from distributed<2.0,>=1.21.8->prefect)
      Downloading https://files.pythonhosted.org/packages/2f/b8/11ec5006d2ec2998cb68349b8d1317c24c284cf918ecd6729739388e4c56/psutil-5.6.1.tar.gz (427kB)
    Collecting certifi>=2017.4.17 (from requests<3.0,>=2.20->prefect)
      Downloading https://files.pythonhosted.org/packages/60/75/f692a584e85b7eaba0e03827b3d51f45f571c2e793dd731e598828d380aa/certifi-2019.3.9-py2.py3-none-any.whl (158kB)
    Collecting chardet<3.1.0,>=3.0.2 (from requests<3.0,>=2.20->prefect)
      Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
    Collecting urllib3<1.25,>=1.21.1 (from requests<3.0,>=2.20->prefect)
      Downloading https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl (118kB)
    Collecting docker-pycreds>=0.4.0 (from docker<4.0,>=3.4.1->prefect)
      Downloading https://files.pythonhosted.org/packages/f5/e8/f6bd1eee09314e7e6dee49cbe2c5e22314ccdb38db16c9fc72d2fa80d054/docker_pycreds-0.4.0-py2.py3-none-any.whl
    Collecting websocket-client>=0.32.0 (from docker<4.0,>=3.4.1->prefect)
      Downloading https://files.pythonhosted.org/packages/29/19/44753eab1fdb50770ac69605527e8859468f3c0fd7dc5a76dd9c4dbd7906/websocket_client-0.56.0-py2.py3-none-any.whl (200kB)
    Collecting cffi!=1.11.3,>=1.8 (from cryptography<3.0,>=2.2.2->prefect)
      Downloading https://files.pythonhosted.org/packages/5b/44/fdae2a8f66af426055f9b6fff0b155217081eddaf08a3df79ca11fe05bda/cffi-1.12.2-cp35-cp35m-manylinux1_x86_64.whl (428kB)
    Collecting asn1crypto>=0.21.0 (from cryptography<3.0,>=2.2.2->prefect)
      Downloading https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl (101kB)
    Collecting typed-ast<1.4.0,>=1.3.1 (from mypy<0.700,>=0.600->prefect)
      Downloading https://files.pythonhosted.org/packages/b1/62/7e0bfe7bb75bf8a41b58ccb7bd3ab3afaec61e06d9abfda113ad0dc227be/typed_ast-1.3.1-cp35-cp35m-manylinux1_x86_64.whl (734kB)
    Collecting heapdict (from zict>=0.1.3->distributed<2.0,>=1.21.8->prefect)
      Downloading https://files.pythonhosted.org/packages/e2/ca/f5feba2f939c97629dbce52a17acc95a0d10256ef620334795379dda8ce6/HeapDict-1.0.0.tar.gz
    Collecting pycparser (from cffi!=1.11.3,>=1.8->cryptography<3.0,>=2.2.2->prefect)
      Downloading https://files.pythonhosted.org/packages/68/9e/49196946aee219aead1290e00d1e7fdeab8567783e83e1b9ab5585e6206a/pycparser-2.19.tar.gz (158kB)
    Building wheels for collected packages: pyyaml, toolz, tornado, psutil, heapdict, pycparser
      Building wheel for pyyaml (setup.py): started
      Building wheel for pyyaml (setup.py): finished with status 'done'
      Stored in directory: /home/user/.cache/pipenv/wheels/ad/da/0c/74eb680767247273e2cf2723482cb9c924fe70af57c334513f
      Building wheel for toolz (setup.py): started
      Building wheel for toolz (setup.py): finished with status 'done'
      Stored in directory: /home/user/.cache/pipenv/wheels/f4/0c/f6/ce6b2d1aa459ee97cc3c0f82236302bd62d89c86c700219463
      Building wheel for tornado (setup.py): started
      Building wheel for tornado (setup.py): finished with status 'done'
      Stored in directory: /home/user/.cache/pipenv/wheels/61/7e/7a/5e02e60dc329aef32ecf70e0425319ee7e2198c3a7cf98b4a2
      Building wheel for psutil (setup.py): started
      Building wheel for psutil (setup.py): finished with status 'done'
      Stored in directory: /home/user/.cache/pipenv/wheels/86/36/f4/e1fc49f198c623b8dc2156e80011a52f19a959d538f6b1a5db
      Building wheel for heapdict (setup.py): started
      Building wheel for heapdict (setup.py): finished with status 'done'
      Stored in directory: /home/user/.cache/pipenv/wheels/40/b9/42/344857b482c954f48bcff6db72d388e30bf2bee4ed14706faa
      Building wheel for pycparser (setup.py): started
      Building wheel for pycparser (setup.py): finished with status 'done'
      Stored in directory: /home/user/.cache/pipenv/wheels/f2/9a/90/de94f8556265ddc9d9c8b271b0f63e57b26fb1d67a45564511
    Successfully built pyyaml toolz tornado psutil heapdict pycparser
    Installing collected packages: marshmallow, marshmallow-oneofschema, six, python-dateutil, pytzdata, pendulum, toolz, pyyaml, heapdict, zict, dask, sortedcontainers, click, tornado, tblib, cloudpickle, msgpack, psutil, distributed, idna, certifi, chardet, urllib3, requests, xxhash, toml, docker-pycreds, websocket-client, docker, pycparser, cffi, asn1crypto, cryptography, pytz, mypy-extensions, typed-ast, mypy, croniter, typing, typing-extensions, prefect
    Successfully installed asn1crypto-0.24.0 certifi-2019.3.9 cffi-1.12.2 chardet-3.0.4 click-7.0 cloudpickle-0.6.1 croniter-0.3.29 cryptography-2.6.1 dask-1.1.4 distributed-1.26.0 docker-3.7.1 docker-pycreds-0.4.0 heapdict-1.0.0 idna-2.7 marshmallow-3.0.0b19 marshmallow-oneofschema-2.0.0b2 msgpack-0.6.1 mypy-0.670 mypy-extensions-0.4.1 pendulum-2.0.4 prefect-0.5.0 psutil-5.6.1 pycparser-2.19 python-dateutil-2.8.0 pytz-2018.9 pytzdata-2018.9 pyyaml-3.13 requests-2.21.0 six-1.12.0 sortedcontainers-2.1.0 tblib-1.3.2 toml-0.10.0 toolz-0.9.0 tornado-6.0.2 typed-ast-1.3.1 typing-3.6.6 typing-extensions-3.7.2 urllib3-1.24.1 websocket-client-0.56.0 xxhash-1.3.0 zict-0.1.4
    
    Adding prefect to Pipfile's [packages]…
    Pipfile.lock not found, creating…
    Locking [dev-packages] dependencies…
    Locking [packages] dependencies…
    
    Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
      First try clearing your dependency cache with $ pipenv lock --clear, then try the original command again.
     Alternatively, you can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
      Hint: try $ pipenv lock --pre if it is a pre-release dependency.
    Could not find a version that matches marshmallow-oneofschema<3.0,>=2.0.0b2
    Tried: 1.0, 1.0.1, 1.0.2, 1.0.3, 1.0.3, 1.0.4, 1.0.4, 1.0.5, 1.0.5
    Skipped pre-versions: 2.0.0b1, 2.0.0b1, 2.0.0b2, 2.0.0b2
    There are incompatible versions in the resolved dependencies.
    
    
    opened by dazzag24 25
  • Test issue for notifications

    Test issue for notifications

    opened by jlowin 22
  • A Pandas DataFrame Result

    A Pandas DataFrame Result

    Thanks for contributing to Prefect!

    Please describe your work and make sure your PR:

    • [x] adds new tests (if appropriate)
    • [x] updates CHANGELOG.md (if appropriate)
    • [x] updates docstrings for any new functions or function arguments, including docs/outline.toml for API reference docs (if appropriate)

    Note that your PR will not be reviewed unless all three boxes are checked.

    What does this PR change?

    This PR implements a new result handler that saves and loads Pandas DataFrames using Panda's built-in tools. It's based on the same handler in my prefect_ds library. Discussion in this Slack thread may also be useful as a reference for why this PR is coming into Prefect looking like it does.

    This PR, in its current form, is not something I would consider mergeable into Prefect, but rather is intended to start the conversation about what modifications would need to be made in order to make it mergeable. There are three main reasons for this:

    1. PandasResultHandler requires the full filepath, including the file name, to be specified. This is in contrast (I believe) to the rest of the file handlers, which have the user specify the directory/bucket/container/etc and then generate the actual filename themselves. My interpretation of this design choice is to make using the result handlers with map easier, because with the full filepath specified a mapped task would just write to the same file for every iteration of the map.
    2. PandasResultHandler has hooks for formatting the filepath based on the inputs to the task, to handle the aforementioned mapping problem. But obviously Prefect does not recognize these hooks. (For reference, to get around this in prefect_ds I subclassed the TaskRunner and created a new state handler. This was a bit of a kludge, and could probably be implemented more elegantly in Prefect proper if that was a direction folks wanted to go.)
    3. PandasResultHandler would add an optional Pandas dependency, and currently that's not included in any of the docs, installation tools, or CI (although I did try to account for it in the importing of the handler itself and in the tests). That's not a particularly challenging thing to resolve, but I wanted guidance on how y'all would like that to be implemented.

    Why is this PR important?

    A lot of data workloads, especially data analysis/modeling workflows, revolve around Pandas DataFrames. Furthermore, when analyzing data it's important to be able to manually inspect and validate what you're dealing with, which is hard to do with the LocalResultHandler writing files to randomly named pickles.

    @cicdw (and/or anyone else) take a look and let me know what you think (no rush through 😄).

    opened by AndrewRook 21
  • Styling Prefect Docs

    Styling Prefect Docs

    What does this PR change?

    • core documentation styling

    Why is this PR important?

    • imported and implemented prefect fonts
    • added favicon
    • updated svg logos
    • migrated and refactored overriding styles to style.styl based on vuepress doc update
    • updated router link + anchor tags to use semantic <button> where appropriate for a11y reasons

    issues to work out

    • [x] login button styling
    • [x] add in dark-language highlight for gQL
    • [x] italicize keywords and comments / outputs
    • [x] source-code text styling for reference docs
    docs 
    opened by itsngansense 21
  • Pass `inputs` through to Result class

    Pass `inputs` through to Result class

    I would like to be able to access a tasks inputs when reading/writing results. This allows properly caching based on the different inputs to a function, for example by templating a filename as experiment;m=5.

    Current behavior

    Please describe how the feature works today The prefect.context is passed to the read/write(exists?) methods of the Results class

    Proposed behavior

    Please describe your proposed change to the current behavior Please pass the inputs also.

    Example

    Please give an example of how the enhancement would be useful This would allow something like the following

    @task(result=GCSResult(bucket='bucket', location='{task_name};m={inputs["multiplier"].txt'))
    def experiment(multiplier=1):
        # do something
        return
    
    enhancement 
    opened by limx0 20
  • Replace Dask with ray, how to do?

    Replace Dask with ray, how to do?

    Replace Dask with ray how to do ?

    https://github.com/ray-project/ray

    Or can use Dask on Ray ?

    https://docs.ray.io/en/master/dask-on-ray.html

    https://www.youtube.com/watch?reload=9&v=kZCmqnBrUp4

    enhancement 
    opened by ggservice007 20
  • PIN 5: Combining Tasks

    PIN 5: Combining Tasks

    PIN-5: Date: 2019-02-20

    Author: Chris White

    Status

    Proposed

    Context

    Imagine the following typical scenario: a data engineer wants to create a Prefect Flow which routinely migrates some data from S3 to Google Cloud Storage (along with other things). In our current framework, we implicitly recommend the user do something like (pseudo-code):

    s3_task = S3Task(..)
    gcs_task = GCSTask(...)
    
    with Flow("migration") as f:
        gcs_task(data=s3_task)
    

    This is OK, but imagine the S3 Task returns 10 GB of data, and the user routinely likes using "checkpointing". In this case, the data coming out of S3 will hit the checkpoint, be shipped off somewhere else (dragging this Flow down), and then have to move around the Dask workers, resulting in large and unnecessary data movement. Moreover, many of these infrastructure / db clients have hooks for large data streams that we can't take advantage of with this setup.

    Another option is for the user to re-implement all the hooks / credentials / etc. for both GCS and S3, resulting in a monster S3toGCSTask. With this pattern, if we have M sources and N sinks, we need to maintain and test M*N different tasks (this is what frameworks like Airflow currently do). We want to avoid this situation. Ideally we should only have to maintain M + N tasks that can be flexibly and powerfully composed in various ways.

    Additionally, it would be nice if users could specify that two tasks should run on the same worker, in the same process, and share memory.

    Decision

    We will implement some sugar which allows users to combine two tasks into a single Task. For example, the imperative version of this might look like (pseudo-code):

    class CombinedTask(Task):                                                             
        def __init__(self, first_task: Task, second_task: Task):                        
            self.first_task = first_task                                             
            self.second_task = second_task                                                 
                                                                                       
        def run(self):                                                                 
            inputs = first_task.run()                                                 
            result = second_task.run(inputs)
    

    along with a functional context manager:

    with one_task():
        second_task(first_task)
    

    Of course, there is some work that needs to be done under the hood to match inputs / outputs, and allow for calling patterns such as

    with one_task():
        second_task(first_task(config="some_setting"), parameter="another_input")
    

    But ultimately, these two tasks would be combined into a single task which is submitted to a single worker.

    How many tasks?

    This PIN proposes we only support combining two tasks, with our target use case being migrating data. Allowing for arbitrary numbers might encourage an anti-pattern (Prefect generally prefers small, modular tasks), and become a headache to maintain (deciding which arguments to a combined task should actually be combined vs. left as standalone tasks will be tricky).

    Consequences

    The largest user-facing consequence is that, if a user uses this pattern, they lose any prefect hooks which may occur between the two tasks, such as trigger checks, notifications, state handlers, etc. In my view, this is perfectly OK in certain situations such as this, where the goal is to move data. If something fails, the data is still sitting in S3, and the user just needs the error to debug.

    Exposing this pattern to users will certainly appease many of the data engineers we've talked to, as well as reduce the load on our system. Additionally, it would allow us to utilize a shared (temporary) filesystem for these connected / combined tasks and connect to different hooks that otherwise wouldn't be available to us.

    opened by cicdw 19
  • Broken docker-compose.yml

    Broken docker-compose.yml

    Description

    prefect server start fails since 0.14.7 release.

    Expected Behavior

    Prefect UI starting.

    Observed behaviour

    Execution fails with

    ERROR: The Compose file './docker-compose.yml' is invalid because:
    services.ui.depends_on contains an invalid type, it should be an array
    services.apollo.depends_on contains an invalid type, it should be an array
    services.towel.depends_on contains an invalid type, it should be an array
    services.graphql.depends_on contains an invalid type, it should be an array
    services.hasura.depends_on contains an invalid type, it should be an array
    Exception caught; killing services (press ctrl-C to force)
    ERROR: The Compose file './docker-compose.yml' is invalid because:
    services.ui.depends_on contains an invalid type, it should be an array
    services.apollo.depends_on contains an invalid type, it should be an array
    services.towel.depends_on contains an invalid type, it should be an array
    services.graphql.depends_on contains an invalid type, it should be an array
    services.hasura.depends_on contains an invalid type, it should be an array
    Traceback (most recent call last):
      File "/home/vincentchery/miniconda3/envs/prefect_test/lib/python3.8/site-packages/prefect/cli/server.py", line 346, in start
        subprocess.check_call(
      File "/home/vincentchery/miniconda3/envs/prefect_test/lib/python3.8/subprocess.py", line 364, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['docker-compose', 'pull']' returned non-zero exit status 1.
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/home/vincentchery/miniconda3/envs/prefect_test/bin/prefect", line 10, in <module>
        sys.exit(cli())
      File "/home/vincentchery/miniconda3/envs/prefect_test/lib/python3.8/site-packages/click/core.py", line 829, in __call__
        return self.main(*args, **kwargs)
      File "/home/vincentchery/miniconda3/envs/prefect_test/lib/python3.8/site-packages/click/core.py", line 782, in main
        rv = self.invoke(ctx)
      File "/home/vincentchery/miniconda3/envs/prefect_test/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/home/vincentchery/miniconda3/envs/prefect_test/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/home/vincentchery/miniconda3/envs/prefect_test/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/home/vincentchery/miniconda3/envs/prefect_test/lib/python3.8/site-packages/click/core.py", line 610, in invoke
        return callback(*args, **kwargs)
      File "/home/vincentchery/miniconda3/envs/prefect_test/lib/python3.8/site-packages/prefect/cli/server.py", line 384, in start
        subprocess.check_output(
      File "/home/vincentchery/miniconda3/envs/prefect_test/lib/python3.8/subprocess.py", line 411, in check_output
        return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
      File "/home/vincentchery/miniconda3/envs/prefect_test/lib/python3.8/subprocess.py", line 512, in run
        raise CalledProcessError(retcode, process.args,
    subprocess.CalledProcessError: Command '['docker-compose', 'down']' returned non-zero exit status 1.
    

    Reproduction

    • Install prefect >= 0.14.7
    • run
    prefect backend server
    prefect server start
    

    Environment

    • Ubuntu 20.04
    • Fresh virtualenv with only prefect installed, versions 0.14.7+ fail, versions <0.14.7 work as expected.

    Solution

    I tested prefect versions 0.14.0 to 0.14.9 and observed that the bug was introduced in release 0.14.7 in src/prefect/cli/docker-compose.yml.

    depends_on should be an array (starting by hyphen "-") as stated in the error logs produced when running prefect server start, and the diff shows that hyphens were removed from depends_on options in 0.14.7.

    See for instance : https://github.com/PrefectHQ/prefect/pull/4041/files#diff-110ca811f8b6466bd56ed8607c3a94de6925d89f71acedafd1ee71c166ab831dR54

    --> Put hyphens back in depends_on options of the docker-compose.yaml :)

    opened by VincentAntoine 18
  • Docs: Orchestration orchestration cloud examples

    Docs: Orchestration orchestration cloud examples

    Summary

    Docs changes directing first time users to Prefect Cloud, and providing more clear differentiation between optimal Prefect Cloud and Prefect Core server use cases.

    Changes

    Edits to the following source documents:

    docs/orchestration/README.md docs/orchestration/getting-started/basic-core-flow.md docs/orchestration/getting-started/quick-start.md docs/orchestration/getting-started/registering-and-running-a-flow.md docs/orchestration/getting-started/set-up.md

    Affects the following published pages:

    https://docs.prefect.io/orchestration/ https://docs.prefect.io/orchestration/getting-started/quick-start.html https://docs.prefect.io/orchestration/getting-started/basic-core-flow.html https://docs.prefect.io/orchestration/getting-started/set-up.html https://docs.prefect.io/orchestration/getting-started/registering-and-running-a-flow.html

    Also moved the Python 3.9 warning so that installation is consistent across both Core and Orchestration docs.

    Misc changes for consistent phrasing and usage across pages.

    Importance

    Using Prefect Cloud is a better first-time experience. Users get a better onramp to all of the features available for orchestrating flows.

    Also make sure to document scenarios where server is optimal: your environment has restricted access to the Internet, or you prefer a self-hosted server for which you want to build your own authentication and scaling mechanism.

    Checklist

    This PR:

    • [ ] adds new tests (if appropriate)
    • [ ] adds a change file in the changes/ directory (if appropriate)
    • [ ] updates docstrings for any new functions or function arguments, including docs/outline.toml for API reference docs (if appropriate)
    opened by tpdorsey 0
  • Add dbt cloud tasks

    Add dbt cloud tasks

    Summary

    This PR adds a new task to trigger a dbt Cloud job run.

    Importance

    This new task enable users to orchestrate dbt Cloud job runs from within Prefect.

    Checklist

    This PR:

    • [x] adds new tests (if appropriate)
    • [x] adds a change file in the changes/ directory (if appropriate)
    • [x] updates docstrings for any new functions or function arguments, including docs/outline.toml for API reference docs (if appropriate)
    opened by AlessandroLollo 0
  • update bigquery dependency to support version 2.0

    update bigquery dependency to support version 2.0

    Summary

    Remove constraint on bigquery dependency that it must be below version 2.0.

    Changes

    My project is now using a newer (2.x) version of Biquery and is incompatible with the constraints in prefect's setup.py if I want the GCP extra.

    No APIs need to be changed according to the migration guide since we are not using any removed/deprecated items. https://googleapis.dev/python/bigquery/latest/UPGRADING.html

    Importance

    Other users probably have encountered this issue or will likely encounter it. The 2.x google bigquery library has been out since September 2020. https://pypi.org/project/google-cloud-bigquery/#history

    Checklist

    This PR:

    • [x] adds new tests (if appropriate)
    • [x] adds a change file in the changes/ directory (if appropriate)
    • [x] updates docstrings for any new functions or function arguments, including docs/outline.toml for API reference docs (if appropriate)
    opened by wangjoshuah 0
  • Adding logo to README

    Adding logo to README

    Summary

    Put the Census Logo in the assets and added it as a link within the README.md making sure that the Tasks remain balanced.

    Changes

    Just the README.md to add the logo.

    Importance

    Clearly documents that Reverse ETL is compatible with Prefect

    Checklist

    This PR:

    • [ ] adds new tests (if appropriate)
    • [ ] adds a change file in the changes/ directory (if appropriate)
    • [ ] updates docstrings for any new functions or function arguments, including docs/outline.toml for API reference docs (if appropriate)
    opened by dflynn20 0
  • Orion: No default database created

    Orion: No default database created

    Description

    In the README, it says the following: Please note steps 1 and 2 are optional. The Orion server will use an in-memory SQLite database by default.

    However, when I try to execute an orion flow, I get the following error message:

    sqlite3.OperationalError: unable to open database file
    
    Detailed error message
    Traceback (most recent call last):
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3240, in _wrap_pool_connect
        return fn()
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 310, in connect
        return _ConnectionFairy._checkout(self)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 868, in _checkout
        fairy = _ConnectionRecord.checkout(pool)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 476, in checkout
        rec = pool._do_get()
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/pool/impl.py", line 256, in _do_get
        return self._create_connection()
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 256, in _create_connection
        return _ConnectionRecord(self)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 371, in __init__
        self.__connect()
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 666, in __connect
        pool.logger.debug("Error on connect(): %s", e)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
        compat.raise_(
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
        raise exception
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
        self.dbapi_connection = connection = pool._invoke_creator(self)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/engine/create.py", line 590, in connect
        return dialect.connect(*cargs, **cparams)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 584, in connect
        return self.dbapi.connect(*cargs, **cparams)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 292, in connect
        await_only(connection),
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 76, in await_only
        return current.driver.switch(awaitable)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 129, in greenlet_spawn
        value = await result
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/aiosqlite/core.py", line 137, in _connect
        self._connection = await future
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/aiosqlite/core.py", line 102, in run
        result = function()
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/aiosqlite/core.py", line 397, in connector
        return sqlite3.connect(loc, **kwargs)
    sqlite3.OperationalError: unable to open database file
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "/Users/marius/Documents/Projekte/Pexon/FMC/prefect-playground/app.py", line 8, in <module>
        state = my_favorite_function()
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/prefect/flows.py", line 227, in __call__
        return enter_flow_run_engine_from_flow_call(self, parameters)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/prefect/engine.py", line 115, in enter_flow_run_engine_from_flow_call
        return anyio.run(begin_run)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/anyio/_core/_eventloop.py", line 56, in run
        return asynclib.run(func, *args, **backend_options)  # type: ignore
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 230, in run
        return native_run(wrapper(), debug=debug)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/asyncio/runners.py", line 44, in run
        return loop.run_until_complete(main)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
        return future.result()
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 225, in wrapper
        return await func(*args)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/prefect/client.py", line 58, in wrapper
        return await fn(*args, **kwargs)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/prefect/engine.py", line 149, in create_then_begin_flow_run
        flow_run = await client.create_flow_run(
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/prefect/client.py", line 351, in create_flow_run
        flow_id = await self.create_flow(flow)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/prefect/client.py", line 189, in create_flow
        response = await self.post("/flows/", json=flow_data.dict(json_compatible=True))
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/prefect/client.py", line 117, in post
        response = await self._client.post(route, **kwargs)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/httpx/_client.py", line 1825, in post
        return await self.request(
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/httpx/_client.py", line 1494, in request
        response = await self.send(
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/httpx/_client.py", line 1586, in send
        response = await self._send_handling_auth(
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/httpx/_client.py", line 1616, in _send_handling_auth
        response = await self._send_handling_redirects(
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/httpx/_client.py", line 1655, in _send_handling_redirects
        response = await self._send_single_request(request, timeout)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/httpx/_client.py", line 1699, in _send_single_request
        ) = await transport.handle_async_request(
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/httpx/_transports/asgi.py", line 157, in handle_async_request
        await self.app(scope, receive, send)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/fastapi/applications.py", line 208, in __call__
        await super().__call__(scope, receive, send)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/starlette/applications.py", line 112, in __call__
        await self.middleware_stack(scope, receive, send)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/starlette/middleware/errors.py", line 181, in __call__
        raise exc
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/starlette/middleware/errors.py", line 159, in __call__
        await self.app(scope, receive, _send)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/starlette/middleware/cors.py", line 84, in __call__
        await self.app(scope, receive, send)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/starlette/exceptions.py", line 82, in __call__
        raise exc
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/starlette/exceptions.py", line 71, in __call__
        await self.app(scope, receive, sender)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/starlette/routing.py", line 656, in __call__
        await route.handle(scope, receive, send)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/starlette/routing.py", line 259, in handle
        await self.app(scope, receive, send)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/starlette/routing.py", line 61, in app
        response = await func(request)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/fastapi/routing.py", line 216, in app
        solved_result = await solve_dependencies(
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/fastapi/dependencies/utils.py", line 521, in solve_dependencies
        solved = await solve_generator(
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/fastapi/dependencies/utils.py", line 442, in solve_generator
        return await stack.enter_async_context(cm)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/contextlib.py", line 575, in enter_async_context
        result = await _cm_type.__aenter__(cm)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/contextlib.py", line 181, in __aenter__
        return await self.gen.__anext__()
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/prefect/orion/api/dependencies.py", line 18, in get_session
        engine = await get_engine(timeout=settings.orion.database.timeout)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/prefect/orion/utilities/database.py", line 115, in get_engine
        await create_db(engine)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/prefect/orion/utilities/database.py", line 747, in create_db
        async with engine.begin() as conn:
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/ext/asyncio/base.py", line 60, in __aenter__
        return await self.start(is_ctxmanager=True)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/ext/asyncio/engine.py", line 586, in start
        await self.conn.start(is_ctxmanager=is_ctxmanager)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/ext/asyncio/engine.py", line 131, in start
        await (greenlet_spawn(self.sync_engine.connect))
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 134, in greenlet_spawn
        result = context.throw(*sys.exc_info())
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/future/engine.py", line 419, in connect
        return super(Engine, self).connect()
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3194, in connect
        return self._connection_cls(self, close_with_result=close_with_result)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 96, in __init__
        else engine.raw_connection()
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3273, in raw_connection
        return self._wrap_pool_connect(self.pool.connect, _connection)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3243, in _wrap_pool_connect
        Connection._handle_dbapi_exception_noconnection(
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2097, in _handle_dbapi_exception_noconnection
        util.raise_(
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
        raise exception
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3240, in _wrap_pool_connect
        return fn()
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 310, in connect
        return _ConnectionFairy._checkout(self)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 868, in _checkout
        fairy = _ConnectionRecord.checkout(pool)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 476, in checkout
        rec = pool._do_get()
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/pool/impl.py", line 256, in _do_get
        return self._create_connection()
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 256, in _create_connection
        return _ConnectionRecord(self)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 371, in __init__
        self.__connect()
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 666, in __connect
        pool.logger.debug("Error on connect(): %s", e)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
        compat.raise_(
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
        raise exception
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
        self.dbapi_connection = connection = pool._invoke_creator(self)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/engine/create.py", line 590, in connect
        return dialect.connect(*cargs, **cparams)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 584, in connect
        return self.dbapi.connect(*cargs, **cparams)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 292, in connect
        await_only(connection),
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 76, in await_only
        return current.driver.switch(awaitable)
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 129, in greenlet_spawn
        value = await result
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/aiosqlite/core.py", line 137, in _connect
        self._connection = await future
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/aiosqlite/core.py", line 102, in run
        result = function()
      File "/Users/marius/miniforge3/envs/prefect-playground/lib/python3.9/site-packages/aiosqlite/core.py", line 397, in connector
        return sqlite3.connect(loc, **kwargs)
    sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) unable to open database file
    (Background on this error at: https://sqlalche.me/e/14/e3q8)
    

    I looked around a little bit and logged the connection url in the get_engine method in this file:

    sqlite+aiosqlite://///Users/marius/.prefect/orion.db
    

    I checked and discovered that there is no .prefect directory in my user directory.

    Reproduction / Example

    I installed prefect and tried to execute this program:

    from prefect import flow
    
    @flow
    def my_favorite_function():
        print("This function doesn't do much")
        return 42
    
    state = my_favorite_function()
    print(state)
    print(state.result())
    

    Environment

    Prefect version: 2.0a3 Python version: Python 3.9.7

    opened by mariushab 4
  • 20201020 airbyte

    20201020 airbyte

    Summary

    This PR contains the AirbyteConnectionTask.

    Changes

    This PR adds a new dependency - responds - for requests mocking.

    Importance

    It adds a new integration for Airbyte

    Checklist

    This PR:

    • [X ] adds new tests (if appropriate)
    • [ X] adds a change file in the changes/ directory (if appropriate)
    • [ X] updates docstrings for any new functions or function arguments, including docs/outline.toml for API reference docs (if appropriate)
    opened by iamniche-too 1
  • Glob Task

    Glob Task

    Summary

    Adds a new task to the files section to receive all file/subdirectory paths of given path within the file system.

    Initial work was introduced in #4218. Additional changes have been made per reviewer's request on initial PR.

    Changes

    Adds new task Glob and unit test coverage for new task.

    Importance

    This file operation doesn't currently exist in the task library.

    Checklist

    This PR:

    • [x] adds new tests (if appropriate)
    • [x] adds a change file in the changes/ directory (if appropriate)
    • [x] updates docstrings for any new functions or function arguments, including docs/outline.toml for API reference docs (if appropriate)
    opened by desertaxle 1
  • WIP: Improve handling of missing context ids

    WIP: Improve handling of missing context ids

    Summary

    Addressing issues in https://github.com/PrefectHQ/prefect/issues/5075

    Install with

    pip install -U git+https://github.com/PrefectHQ/[email protected]#egg=prefect
    

    Changes

    • Improves the error message when a task id or flow run id is missing from the context when initializing a task run
    opened by madkinsz 1
  • Task missing UUID in `get_or_create_task_run_info` in 0.15.5+

    Task missing UUID in `get_or_create_task_run_info` in 0.15.5+

    Users upgrading to 0.15.5+ are encountering empty task id or flow run id string task errors during flow runs.

    For example:

    prefect.exceptions.ClientError: [{'path': ['get_or_create_task_run_info'], 'message': 'Expected type UUID!, found ""; Could not parse UUID: '
    

    If reporting a reproduction, please include:

    • Does this reproduce on 0.15.5?
    • Does this reproduce on 0.15.6 or 0.15.7?
    • What agent are you using?
    • What executor are you using?

    A debug branch that can help us determine the source of the issue exists at https://github.com/PrefectHQ/prefect/pull/5076

    This may be related to https://github.com/PrefectHQ/prefect/pull/4930 and has been reported in Prefect Server and Cloud.

    See previous discussion at

    • #4687
    • https://prefect-community.slack.com/archives/CL09KU1K7/p1634853488119200
    • https://prefect-community.slack.com/archives/CL09KU1K7/p1632735840323400
    opened by zangell44 1
  • Fix RunNamespacedJob Logging Issue

    Fix RunNamespacedJob Logging Issue

    Summary

    The current RunNamespacedJob logging function on UI is broken as reported in issue: #4675 .

    As described from user @arodus's analysis, the issue is because of the missing context for the ReadNamespacedPodLogs task running from the Flow.

    I removed the with ThreadPoolExecutor() as pool: method from it and tested in our project, the logging function is working as expected now:

    image

    Importance

    Blocking users from using RunNamespacedJob to properly log the k8s pods output.

    Checklist

    This PR:

    • [ ] adds new tests (if appropriate)
    • [ ] adds a change file in the changes/ directory (if appropriate)
    • [ ] updates docstrings for any new functions or function arguments, including docs/outline.toml for API reference docs (if appropriate)
    opened by tyun08 1
Releases(0.15.7)
  • 0.15.6(Sep 21, 2021)

    Enhancements

    • Improve setting the Azure storage connection string - #4955
    • Allow disabling retries for a task with max_retries=0 when retries are globally configured - #4971

    Task Library

    • Allow Exasol Tasks to handle Prefect Secrets directly - #4436
    • Adding Census Syncs to the task library - #4935

    Fixes

    • Fix bug where LocalDaskExecutor did not respond to a PrefectSignal - #4924
    • Fix PostgresFetch with headers for one row - #4968
    • Fix bug where apply_map could create acyclic flows - #4970

    Contributors

    Source code(tar.gz)
    Source code(zip)
  • 0.15.5(Sep 2, 2021)

    Released on September 2, 2021.

    Features

    • Python 3.9 docker images are now published - #4896

    Enhancements

    • Add --expose flag to prefect server cli to make the Core server and UI listen to all interfaces - #4821
    • Pass existing/local environment variables to agentless flow runs - #4917
    • Add --idempotency-key to prefect run - #4928
    • Add support for larger flow registration calls - #4930
    • Ignore schedules by default for CLI flow runs and add flag to run based on schedule for local only runs #4817

    Task Library

    • Feature: Added SnowflakeQueryFromFile task #3744
    • Enhancement: Log boto exceptions encountered in the in AWS BatchSubmit task - #4771
    • Breaking: Legacy Dremio authentication has been updated to the new pattern in DremioFetch - #4872
    • Fix: Use runtime arguments over init arguments instead of ignoring them for MySQL Tasks - #4907

    Fixes

    • Adjust log limits to match backend logic for better UX - #4900
    • Fix use of marshmallow.fields.Dict to use keys as a kwarg rather than key. - #4903
    • API server settings are passed correctly to task workers when using Prefect Server - #4914
    • Do not attempt attempt to set host_gateway if using an unsupported Docker Engine version - #4809
    • Ignore jobs without a flow_run_id label in KubernetesAgent.manage_jobs - #4934

    Breaking Changes

    • Services run by prefect server cli are now local by default (listen to localhost instead of 0.0.0.0); use --expose if you want to connect from a remote location - #4821
    • The changes in flow registration require Prefect Server 2021.09.02. Prefect Server will need to be upgraded before flows can be registered from this version - #4930

    Contributors

    Source code(tar.gz)
    Source code(zip)
  • 0.15.4(Aug 17, 2021)

    Changelog

    0.15.4

    Released on August 17, 2021.

    Docs

    • Add a getting started section with a quick start guide for both core and orchestration sections - #4734

    Enhancements

    • Expose Snowflake cursor type to SnowflakeQuery task arguments #4786
    • Add ability to use threaded flow heartbeats - #4844
    • Improve behavior when API rate limits are encountered - #4852
    • Allow custom git clone url for Git storage - #4870
    • Add on_worker_status_changed callback to the DaskExecutor - #4874
    • Add --agent-config-id to prefect agent <kubernetes|local> install - #4876

    Task Library

    • Add new prometheus task to push to gateway - #4623

    Fixes

    • Fix binding of named volumes to flow containers with Docker agent - #4800
    • Fix ImportError typo in dropbox module - #4855
    • Fix default safe char for gitlab storage repo path - #4828

    Contributors

    Source code(tar.gz)
    Source code(zip)
  • 0.15.3(Jul 27, 2021)

  • 0.15.2(Jul 21, 2021)

    Enhancements

    • Allow CLI registration of flows without starting their schedule prefect register --no-schedule - #4752
    • Add host_config to DockerRun to expose deeper settings for Docker flow runs - #4733
    • Enable loading additional repository files with Git storage - #4767
    • Update flow run heartbeats to be robust to exceptions - #4736
    • Allow prefect build/register paths to contain globs for recursion - #4761

    Fixes

    • Fix duplicate task runs in FlowRunView.get_all_task_runs - #4774
    • Fix zombie processes from exited heartbeats - #4733
    • Missing auth_file directory is created when saving credentials - #4792

    Task Library

    • Add VaultSecret task for retrieving secrets from private Vault instances - #4656

    Contributors

    Source code(tar.gz)
    Source code(zip)
  • 0.15.1(Jul 12, 2021)

    Enhancements

    • Add documentation for querying role and membership info - #4721
    • Checkpoint task results when SUCCESS is raised - #4744

    Fixes

    • Fix loading of PREFECT__CLOUD__API_KEY environment variable when starting agents - #4751
    • Fix bug where the tenant could not be inferred during flow runs while using token auth - #4758
    • Fix bug where an agent using token auth could clear the tenant from disk - #4759
    Source code(tar.gz)
    Source code(zip)
  • 0.15.0(Jul 1, 2021)

    0.15.0

    Released on July 1, 2021.

    Features

    • Add objects for inspecting flows, flow runs, and task runs without writing queries - #4426
    • Rehaul prefect run CLI for executing flows locally and with agents - #4463
    • Add flow run tasks to simplify flow run result passing - #4563
    • Add agentless execution for flow runs - #4589
    • Add prefect auth create-key to create API keys - #4643
    • Add prefect auth list-keys to list API key metadata - #4643
    • Add prefect auth revoke-key to revoke an API key - #4643
    • Add prefect auth status command to see the state of your authentication - #4643

    Enhancements

    • Improve flow run documentation with new dedicated section - #4492
    • Update Client to support API keys - #4643
    • Add API key support to prefect auth login/logout/switch-tenants - #4643
    • API keys can be configured in the Prefect config - #4643
    • Change SendGrid SendEmail task to use secret value - #4669
    • Add TenantView for retrieval of tenant information - #4676
    • Add custom rbac documentation - #4696
    • Exit with non-zero status on flow run failure when watched - #4709
    • Display return code on local agent flow process failure - #4715

    Task Library

    • Add KafkaBatchConsume and KafkaBatchProduce tasks - #4533

    Fixes

    • Fix cleanup issue with Git storage on Windows - #4665
    • Pass API keys as tokens for compatibility when creating flow run environments - #4683
    • Fix missing event timestamp attribute errors in K8s agent - #4693
    • Fix backwards compatibility for flows without a terminal_state_handler - #4695
    • Raise a better exception when a task run result type is not set in TaskRunView.get_result() - #4708

    Deprecations

    • Deprecate prefect auth create-token - #4643
    • Deprecate prefect auth list-tokens - #4643
    • Deprecate prefect auth revoke-token - #4643
    • Deprecate setting auth tokens in the Prefect config - #4643
    • prefect.utilities.exceptions has been deprecated in favor of prefect.exceptions - #4664

    Breaking Changes

    • Remove deprecated prefect.environment.storage module
    • Remove deprecated DockerAgent kwarg network
    • Remove deprecated kubernetes.ResourceManager class
    • Remove deprecated prefect agent start/install <agent-type> commands
    • Remove deprecated prefect agent local start flag --storage-labels
    • Remove deprecated DroboxDownload task kwarg access_token_secret
    • Remove deprecated GCS... tasks kwarg encryption_key_secret
    • Remove deprecated prefect.tasks.google module
    • Remove deprecated prefect.tasks.secret.Secret class
    • Remove deprecated Scheduler serializers for Prefect <0.6.0
    • Remove deprecated RunGreatExpectionsCheckpoint task
    • Remove deprecated OneTimeSchedule and UnionSchedule classes
    • Remove deprecated flow run tasks ending in Task
    • Remove deprecated prefect.utilities.tasks.unmapped; moved to prefect.utilities.edges.unmapped
    • Prefect state signals now inherit from BaseException to prevent accidental capture - #4664
    • TaskTimeoutError has been replaced with TaskTimeoutSignal - #4664
    • VersionLockError has been replaced with VersionLockMismatchSignal - #4664

    Contributors

    Source code(tar.gz)
    Source code(zip)
  • 0.14.22(Jun 16, 2021)

    0.14.22

    Released on June 15, 2021.

    Enhancements

    • Use functools.update_wrapper for FunctionTask - #4608
    • Add ability to merge reference tasks when combining two flows - #4644
    • Add client side check for key value size - #4655
    • Ensure stack traces are included in logs during task run exceptions - #4657
    • Add poll_interval parameter to StartFlowRun to define the polling interval when waiting for the flow run to finish - #4641
    • Add ability to set task timeouts with timedelta objects - #4619

    Fixes

    • Add ssh documentation - #4539

    Contributors

    Source code(tar.gz)
    Source code(zip)
  • 0.14.21(Jun 2, 2021)

    Features

    • Add interface for backend key-value metadata store - #4499

    Enhancements

    • Keep intermediate docker layers when using Docker Storage to improve caching - #4584

    Fixes

    • Fix possible race condition in LocalResult directory creation - #4587
    • Use absolute paths when registering flows in prefect register - #4593
    • Propagate storage labels (e.g. hostname label on Local storage) when registering flows with prefect register - #4593
    • Fix small-flow parallelism issues with multiprocess LocalDaskExecutor - #4602
    • Cleanly handle unpicklable exceptions in tasks - #4605

    Contributors

    Source code(tar.gz)
    Source code(zip)
  • 0.14.20(May 25, 2021)

    Changelog

    0.14.20

    Released on May 25, 2021.

    Enhancements

    • Refactor Agent base class for readability - #4341
    • Display the agent config id on agent startup if set - #4524
    • Add debug logs during agent auth verification - #4547
    • Sending logs to Cloud can be globally disabled via config in addition to the agent flag - #4487

    Task Library

    • Enable sending attachments with emails in the EmailTask - #4457
    • Add Google Cloud Platform GCPSecret task - #4561

    Fixes

    • Fix import_object handling of submodules that are not attributes - #4513
    • Fix DockerStorage building with python slim image - #4523
    • Gracefully handle events with missing timestamps in K8s agent - #4544
    • Fix bug where agent uses originally scheduled start time instead of latest state time - #4568

    Deprecations

    • logging.log_to_cloud has been deprecated in favor of cloud.send_flow_run_logs - #4487

    Contributors

    Source code(tar.gz)
    Source code(zip)
  • 0.14.19(May 12, 2021)

  • 0.14.18(May 11, 2021)

    Changelog

    0.14.18

    Released on May 11, 2021.

    Enhancements

    • Add an image_pull_policy kwarg to KubernetesRun - #4462
    • Allow prefect build/register --module to accept full import path to flow - #4468
    • Add hello world flow to prefect module - #4470
    • Set docker container names to flow run names when running with the docker agent - #4485

    Task Library

    • Add basic implementation of SendGrid to Task Library - #4450
    • Log link to flow run in StartFlowRun - #4458
    • Allow passing in io.BytesIO to GCSUpload - #4482

    Fixes

    • Fix logging errors within BigQuery task - #4419
    • Remove unnecessary docker interface detection in Docker agent - #4446
    • Upgrade kubernetes package requirement upper limit - #4452
    • Fix Prefect server startup check for custom server port - #4501

    Contributors

    Source code(tar.gz)
    Source code(zip)
  • 0.14.17(Apr 27, 2021)

    Features

    • Add git storage - #4418

    Enhancements

    • Add test coverage for threaded LocalDaskExecutor timeouts - #4217
    • Add environment variable support to UniversalRunConfig - #4383
    • Adds column name fetching to PostgresFetch task - #4414
    • Allow external Postgres with prefect server start command - #4424

    Fixes

    • Pass reference tasks states instead of task states to terminal_state_handler - #4409
    • Check for AWS_RETRY_MODE variable before setting default in ECSAgent - #4417
    • Fixed bug from Flow.copy() not copying the slugs dictionary - #4435
    • Fix compatibility with PyGithub >= 1.55 - #4440

    Contributors

    Source code(tar.gz)
    Source code(zip)
  • 0.14.16(Apr 14, 2021)

    Changelog

    April 14, 2021

    Released on April 14, 2021.

    Enhancements

    • Added support for Bitbucket cloud into Bitbucket storage class #4318
    • Update docs for API keys - #4313
    • Add option to disable the deletion of finished Prefect jobs in the Kubernetes agent - #4351
    • Improve messaging when a flow is skipped during registration - #4373
    • Display a more helpful error when calling a task outside a flow context - #4374
    • Lower required docker-compose version and add note to docs - #4396
    • Increase healthcheck intervals for Prefect Server - #4396
    • Allow the ShellTask to be used on win32 - #4397

    Fixes

    • Fix ShellTask docstring - #4360
    • Fix incorrect unused task tracking - #4368
    • Add error handling to timeouts that fail during result pickling or passing - #4384

    Contributors

    Source code(tar.gz)
    Source code(zip)
  • 0.14.15(Apr 5, 2021)

    Names are hard and stressful.

    Changelog

    April 05, 2021

    Released on April 5, 2021.

    Enhancements

    • Add terminal flow state handler override - #4198
    • When manually set, prefect.context.date will be used to determine dependent values - #4295
    • prefect.context.date will be cast to a DateTime object if given a parsable string - #4295
    • Expand logging for DaskExecutor, including the cluster dashboard address (if available) - #4321
    • Add ability to stream ShellTask logs with level INFO - #4322
    • Add architecture diagram to docs - #4187
    • Speed up flow validation logic - #4347
    • Pin dask upper package versions - #4350

    Task Library

    • Add first basic implementation of Soda SQL Scan task to run DQ tests using soda-sql library
    • Add new task KubernetesSecret - #4307

    Fixes

    • Fix DatabricksRunNow task attribute override behavior - #4309
    • Use default flow labels when triggering flow runs from CLI - #4316
    • Improve ECS agent error messages, and fix bug that prevented using ECS agent on Fargate with ECR - #4325
    • Support imports from local directory when registering/building flows via CLI - #4332
    • Speedup flushing of logs to cloud/server on flow run shutdown, avoiding lost logs on platforms that SIGKILL the process after a short period - #4334

    Contributors

    Source code(tar.gz)
    Source code(zip)
  • 0.14.14(Mar 26, 2021)

    Changelog

    Released on March 25, 2021.

    Task Library

    • Ensures Snowflake Query Task output is serializable [#3744] (https://github.com/PrefectHQ/prefect/issues/3744)

    Fixes

    • Always load task run info prior to running a task - #4296
    Source code(tar.gz)
    Source code(zip)
  • 0.14.13(Mar 24, 2021)

    Changelog

    Released on March 24, 2021.

    Features

    • Add new improved prefect register CLI command. This command supports registering multiple flows at once (through multiple file paths or directories), and also includes a new --watch flag for watching files and automatically re-registering upon changes. - #4256
    • New prefect build CLI command for building flows. Artifacts produced by this command can then be used by prefect register to register flows without requiring the source. - #4282

    Enhancements

    • Use explicit exception chaining #3306
    • Add as_bytes option to S3Download task - #4238
    • Improve error message when a loaded flow doesn't match the version stored in Prefect Cloud/Server - #4259
    • Support setting flow run labels from cli - #4266
    • Support setting default image in --job-template/--task-definition in Kubernetes/ECS agents - #4270

    Task Library

    • Adds logging of cell outputs to Jupyter task - [#4265] (https://github.com/PrefectHQ/prefect/issues/4265)
    • Add user and password as runtime parameters to Exasol tasks - #4268

    Fixes

    • Fix bug where sometimes the global prefect.context wouldn't be respected during a flow run - #4287

    Deprecations

    • Deprecate the old prefect register flow CLI command in favor of prefect register - #4256
    • Deprecate user and password arguments to Exasol task constructors in favor of runtime parameters - #4268

    Contributors

    Source code(tar.gz)
    Source code(zip)
  • 0.14.12(Mar 10, 2021)

    Changelog

    0.14.12

    Released on March 10, 2021.

    Enhancements

    • Upgrade hasura to 1.3.3 in Prefect Server - #4126
    • Add --docker-client-timeout flag to docker agent, for configuring the timeout for all docker API requests - #4232
    • Make --slug flag optional in prefect server create-tenant - #4240
    • Move command, environment, cpu, memory, execution_role_arn, and task_role_arn configuration for ECS tasks from definition time to run time in the ECS agent - #4211
    • Register (and deregister) a new task definition for every flow run in ECS agent - #4211

    Task Library

    • Adds new filesystem Copy and Remove tasks - #4202

    Fixes

    • Don't forward nout to mapped tasks - #4206
    • Fix Task signature generation in the presence of with variadic kwargs - #4235
    • Ensure Flow is serializable using pickle - #4209

    Contributors

    Source code(tar.gz)
    Source code(zip)
  • 0.14.11(Mar 3, 2021)

    Changelog

    0.14.11

    Released on March 3, 2021.

    Features

    • Add command prefect server config to output configured docker-compose yaml - #4176

    Enhancements

    • Add project_name and project_id to context during Cloud/Server flow runs - #4083
    • Better error message when flow not found in file - #4182
    • Improve generated names for GetItem tasks - #4183
    • Add base_url option to GitHub storage - #4194

    Task Library

    • Add filehandling tasks Move, Unzip, Zip - #4131
    • Add msg_plain, email_to_cc, and email_to_bcc options to EmailTask - #4157
    • Add jar_params option to DatabricksRunNow task - #4157

    Fixes

    • Make task slug generation robust to modifying existing task names - #4189
    • Forward client_options to S3Result from S3 storage - #4195

    Contributors

    Source code(tar.gz)
    Source code(zip)
  • 0.14.10(Feb 23, 2021)

    Changelog

    0.14.10

    Released on February 23, 2021.

    Fixes

    • Dynamically import compression libraries for CompressedSerializer - #4150
    • Support passing environment variables containing = through agent CLI --env flag - #4160
    Source code(tar.gz)
    Source code(zip)
  • 0.14.9(Feb 16, 2021)

    Changelog

    0.14.9

    Released on February 16, 2021.

    Enhancements

    • Add CompressedSerializer class - #4063
    • Allow Client timeout seconds to be configurable through configuration - #4118

    Task Library

    • Update FilterTask to allow logging the filtered output via a function - #4121
    • Add FivetranSyncTask, to manage your Fivetran connector sync process - #4116

    Fixes

    • Reduce the number of boto3 clients created across Prefect when interacting with AWS services - #4115

    Deprecations

    • Deprecate the use_session argument in all AWS-related components - #4115

    Contributors

    Source code(tar.gz)
    Source code(zip)
  • 0.14.8(Feb 12, 2021)

    Changelog

    0.14.8

    Released on February 11, 2021.

    Enhancements

    • Add option to provide version group ID to prefect run flow CLI command - #4100

    Fixes

    • Fix bug in agent healthcheck route that was introduced in 0.14.7 - #4109
    Source code(tar.gz)
    Source code(zip)
  • 0.14.7(Feb 10, 2021)

    Changelog

    0.14.7

    Released on February 10, 2021.

    Enhancements

    • Support multiple docker networks with Docker Agent - #3986
    • Add healthchecks to prefect server - #4041
    • Raise custom TimeoutError for task timeouts to allow for more granular user control - #4091
    • Add access_token_secret to GitHub, GitLab, and Bitbucket storage, making the Prefect secret containing an access token to these services configurable - #4059
    • Add --skip-if-flow-metadata-unchanged to prefect register flow CLI command that avoids bumping flow version if flow metadata has not changed - #4061
    • Add --skip-if-exists to prefect create project CLI command that safely skips if the project has already been created - #4061
    • Add new Module storage class, for referencing flows importable from a Python module - #4073
    • Drop resource limits from manifest generated using prefect agent kubernetes install - #4077

    Task Library

    • Add new tasks for communication with an Exasol database - #4044

    Fixes

    • Fix task decorator chaining by using inspect.unwrap instead of __wrap__ - #4053
    • Forward Prefect backend type to deployed flow runs, ensuring backend-specific logic functions properly - #4076
    • Patch around bug in dask's multiprocessing scheduler introduced in Dask 2021.02.0 - #4089

    Deprecations

    • Docker agent network kwarg deprecated in favor of networks - #3986

    Breaking Changes

    • Remove unused Storage.get_env_runner method - #4059
    • Remove private utilities in prefect.utilities.git - #4059

    Contributors

    Source code(tar.gz)
    Source code(zip)
  • 0.14.6(Feb 2, 2021)

    Changelog

    0.14.6

    Released on February 2, 2021.

    Enhancements

    • Add option to provide flow ID to run flow CLI command - #4021
    • Flow name and project are no longer required options when calling run flow CLI command - #4021

    Task Library

    • Add GCSBlobExists which checks for the existence of an object in a given GCS bucket - #4025
    • Use boto3 session in S3Upload and S3Download tasks, to ensure thread-safe execution - #3981

    Fixes

    • Fix issue with fixed duration Paused states not resuming properly - #4031

    Contributors

    Source code(tar.gz)
    Source code(zip)
  • 0.14.5(Jan 26, 2021)

    Changelog

    0.14.5

    Released on January 26, 2021.

    Enhancements

    • S3 storage now logs ETag, LastModified timestamp, and VersionId (if present) when loading a flow - #3995
    • GitHub storage now logs the commit sha used when loading a flow - #3998
    • GitHub storage now loads from a repo's default branch, allowing default branch names other than 'master' - #3998
    • Improve error message when Secrets are missing with Server - #4003
    • Better error message when passing parameters to StartFlowRun constructor - #4008
    • Add warning if user-defined class shadows an attribute used by the base class - #4011
    • Add support for EXTRA_PIP_PACKAGES environment variable in prefecthq/prefect images, simplifying installation of dependencies during development - #4013
    • Add execution role arn parameter to ecs run config and agent - #4015

    Task Library

    • Add ConnectGetNamespacedPodExec task which runs an exec command in provided pod container - #3991
    • Ensure connection secrets can be passed to Databricks tasks at runtime - #4001

    Fixes

    • Fix Agent registration possibly skipping on server connection issues - #3972
    • GCSUpload task now explicitely fails when ran on non-supported types - #3978
    • Make logging to Prefect cloud more robust in the presence of errors or process shutdown - #3989
    • Handle setting state for missing flow runs in Kubernetes agent resource management - #4006

    Contributors

    Source code(tar.gz)
    Source code(zip)
  • 0.14.4(Jan 19, 2021)

    Changelog

    0.14.4

    Released on January 19, 2021.

    Enhancements

    • Retry on additional status codes - #3959
    • Rerun secret tasks on flow-run restart - #3977

    Task Library

    • Stream log output from Kubernetes RunNamespacedJob - #3715
    • Add ReadNamespacedPodLogs which reads or streams logs from Kubernetes pod - #3715
    • Add SQL Server task to query SQL Server databases - #3958
    • Add chunking to GCP storage tasks - #3968

    Fixes

    • Properly handle NotImplementedError exceptions raised by a result's serializer - #3964
    • Fix support for storing multiple flows in a single script in storage - #3969
    • Fix regression in apply_map which prevented use in case/resource_manager blocks - #3975

    Contributors

    Source code(tar.gz)
    Source code(zip)
  • 0.14.3(Jan 13, 2021)

    Changelog

    0.14.3

    Released on January 13, 2021.

    Enhancements

    • Better errors/warnings when flow fails to load in execution environment - #3940

    Task Library

    • Add an Asana task to add tasks to an asana project - #3935

    Fixes

    • Fix prefect server start failure when given a custom graphql host port - #3933
    • Fix Kubernetes Agent attempting to report container info for failed pods when no container statuses are found - #3941
    • Avoid race condition when creating task run artifacts for mapped tasks - #3953
    • Propogate agent labels info to k8s flow runs, to match other agent behavior - #3954
    Source code(tar.gz)
    Source code(zip)
  • 0.14.2(Jan 6, 2021)

    Changelog

    Released on January 6, 2021.

    Features

    • Support for specifying run_config for an individual flow run - #3903

    Enhancements

    • Allow the usage of a profile_name on get_boto_client - #3916
    • Support executing Prefect agents/flows without having the prefect CLI on path - #3918

    Task Library

    • Add support for specifying a run_config in StartFlowRun - #3903
    • Task to add Trello card for task library - #3910

    Fixes

    • Remove unused description field on Task serializer - #3917
    • Fix edge case in apply_map that resulted in cycles in the Flow graph - #3920
    • Support storing multiple local flows with the same name when using Local storage - #3923
    • Fix bug in prefect.context contextmanager that resulted in context fields reverting to their initially configured values - #3924

    Contributors

    Source code(tar.gz)
    Source code(zip)
  • 0.14.1(Dec 29, 2020)

    Changelog

    0.14.1

    Released on December 29, 2020.

    Enhancements

    • Make setup method optional for resource_manager tasks - #3869
    • Add labels to all containers managed by the docker agent - #3893
    • Add prefect server stop command for stopping the server - #3899
    • Add --detach to prefect server start for running the server in the background - #3899

    Fixes

    • Add support for google-cloud-storage < 1.31.0 - #3875
    • Fix use of imagePullSecrets/serviceAccountName in k8s agent - #3884
    • Fix read_bytes_from_path to work properly with S3 - #3885
    • Change default idempotency_key in StartFlowRun to use task_run_id instead of flow_run_id - #3892
    Source code(tar.gz)
    Source code(zip)
  • 0.14.0(Dec 16, 2020)

    Changelog

    0.14.0

    Released on December 16, 2020.

    Features

    • New flow run configuration system based on RunConfig types, see here for more info

    Enhancements

    • Kubernetes Agent now reports events for pending pods created by prefect jobs - #3783
    • Using --rbac for Kubernetes Agent install command now includes the events resource - #3783
    • Add orchestration-based dependencies to the prefecthq/prefect Docker image - #3804
    • Add a slimmed down prefecthq/prefect:core Docker image that only contains base dependencies - #3804
    • Docker storage now installs all orchestration-based dependencies when using default image - #3804
    • Add warning on flow registration if flow.executor is set but the flow is using the legacy flow.environment configuration system - #3808
    • Echoing prefect config file as JSON to be able to parse it with jq in the terminal - #3818
    • Produce artifact for RunGreatExpectationsValidation even if validation fails - #3829
    • execute flow-run command now sends flow run log in the case of an error - #3832
    • Changed name of logs raised by the Kubernetes Agent if they stem from infrastructure events - #3832
    • Add tini to the official Prefect docker images - #3839
    • Remove task run level heartbeats for performance - #3842

    Task Library

    • Fix mising job_id in DatabricksRunNow task initialization - #3793

    Fixes

    • Fix Azure result byte decoding of blob data - #3846
    • Prefect kubernetes agent no longer relies on existence of any fields in configured Kubernetes Job Template - #3805
    • Accept old envvar style configuration for Kubernetes agent for --service-account-name/--image-pull-secrets options - #3814
    • Pass as_user=False when using client.get_cloud_url in StartFlowRun - #3850
    • Fix AWS boto3 utility passing duplicate kwargs to client initialization - #3857

    Deprecations

    • Storage classes have been moved from prefect.environments.storage to prefect.storage, the old import paths have been deprecated accordingly - #3796
    • Executor classes have been moved from prefect.engine.executors to prefect.executors, the old import paths have been deprecated accordingly - #3798
    • Deprecated use of storage_labels boolean kwarg on local agent - #3800
    • Deprecated use of --storage-labels option from agent start CLI command - #3800
    • Deprecates all Environment classes - users should transition to setting flow.run_config instead of flow.environment - #3811
    • Deprecate the Fargate Agent in favor of the ECS Agent - #3812

    Breaking Changes

    • Using in-cluster installs of the Kubernetes Agent now requires RBAC for the events resource - #3783
    • Removed setting of default labels on storage objects and the local agent - #3800
    • Remove deprecated RemoteEnvironment/RemoteDaskEnvironment - #3802
    • Remove deprecated executor_kwargs argument to KubernetesJobEnvironment/FargateTaskEnvironment - #3802
    • Remove deprecated prefect run cloud/prefect run server CLI commands - #3803
    • Remove deprecated prefect execute cloud-flow CLI command - #3803
    • Stop building the prefecthq/prefect:all_extras image and switch flow deployment default to using prefecthq/prefect:{core_version} - #3804
    • Flows now use RunConfig based deployments by default - legacy Environment based deployments are now opt-in only - #3806
    • Remove deprecated prefect.contrib module - #3813
    • Remove all references to result handlers and safe results - #3838
    • Remove option to enable deprecated Kubernetes resource manager in agent install CLI command - #3840

    Contributors

    Source code(tar.gz)
    Source code(zip)
Another Scheduler is a Kubernetes controller that automatically starts, stops, or restarts pods from a deployment at a specified time using a cron annotation.

Another Scheduler Another Scheduler is a Kubernetes controller that automatically starts, stops, or restarts pods from a deployment at a specified tim

Diego Najar 47 Oct 13, 2021
Python-Repeated-Timer is an open-source & highly performing timer using only standard-libraries.

Python Repeated Timer Python-Repeated-Timer is an open-source & highly performing timer using only standard-libraries.

TACKHYUN JUNG 3 Sep 8, 2021
A powerful workflow engine implemented in pure Python

Spiff Workflow Summary Spiff Workflow is a workflow engine implemented in pure Python. It is based on the excellent work of the Workflow Patterns init

Samuel 1k Oct 20, 2021
Remote task execution tool

Gunnery Gunnery is a multipurpose task execution tool for distributed systems with web-based interface. If your application is divided into multiple s

Gunnery 730 Oct 16, 2021
A simple scheduler tool that provides desktop notifications about classes and opens their meet links in the browser automatically at the start of the class.

This application provides desktop notifications about classes and opens their meet links in browser automatically at the start of the class.

Anshit 13 Mar 23, 2021
A calendaring app for Django. It is now stable, Please feel free to use it now. Active development has been taken over by bartekgorny.

Django-schedule A calendaring/scheduling application, featuring: one-time and recurring events calendar exceptions (occurrences changed or cancelled)

Tony Hauber 796 Oct 21, 2021
Python job scheduling for humans.

schedule Python job scheduling for humans. Run Python functions (or any other callable) periodically using a friendly syntax. A simple to use API for

Dan Bader 9.1k Oct 23, 2021
Crontab jobs management in Python

Plan Plan is a Python package for writing and deploying cron jobs. Plan will convert Python code to cron syntax. You can easily manage you

Shipeng Feng 1.1k Aug 22, 2021