Metaflow is a human-friendly Python/R library that helps scientists and engineers build and manage real-life data science projects

Overview

Metaflow_Logo_Horizontal_FullColor_Ribbon_Dark_RGB

Metaflow

Metaflow is a human-friendly Python/R library that helps scientists and engineers build and manage real-life data science projects. Metaflow was originally developed at Netflix to boost productivity of data scientists who work on a wide variety of projects from classical statistics to state-of-the-art deep learning.

For more information, see Metaflow's website and documentation.

Getting Started

Getting up and running with Metaflow is easy.

Python

Install metaflow from pypi:

pip install metaflow

and access tutorials by typing:

metaflow tutorials pull

R

Install Metaflow from github:

devtools::install_github("Netflix/metaflow", subdir="R")
metaflow::install()

and access tutorials by typing:

metaflow::pull_tutorials()

Get in Touch

There are several ways to get in touch with us:

Contributing

We welcome contributions to Metaflow. Please see our contribution guide for more details.

Code style

We use black as a code formatter. The easiest way to ensure your commits are always formatted with the correct version of black it is to use pre-commit: install it and then run pre-commit install once in your local copy of the repo.

Comments
  • ERROR: Encountered corrupt package tarball

    ERROR: Encountered corrupt package tarball

    Hi,

    My step works flawlessly in local mode, but when I tried it with batch mode it failed with the message: /bin/sh: 1: metaflow_CleanFlow_linux-64_54816c55859cfd0f8c3c9b2e51678ce87bc33a38/bin/python: not found To understand what might be the reason, I locally ran a docker image(python3:6) and ran all the commands that run on the batch side.

    I noticed when conda creating the environment, some of the packages inside pkgs folder were failing to install. I digged deeper into it and noticed that somehow a few tarballs were not fully copied(~80%) to s3 bucket in the beginning and therefore they were incomplete. I manually downloaded those tarballs and they all started to work fine. What might be the reason for these incomplete tarball uploads to s3 bucket?

    Computer: Mac OSX: 10.15.1 Conda: Anaconda 4.7.12 Metaflow: 2.0.1 List of the tarballs that have failed: chardet-3.0.4-py36_1003.tar.bz2, six-1.14.0-py36_0.tar.bz2, setuptools-45.1.0-py36_0.tar.bz2, pip-20.0.2-py36_1.tar.bz2

    Example error message: ERROR: Encountered corrupt package tarball at /root/.aws/metaflow/conda/pkgs/setuptools-45.1.0-py36_0.tar.bz2. Conda has left it in place. Please report this to the maintainers of your package. For the defaults channel, please report to https://github.com/continuumio/anaconda-issues

    bug enhancement 
    opened by abaspinar 30
  • Support for another public cloud - Microsoft Azure

    Support for another public cloud - Microsoft Azure

    Currently, Metaflow is set up to work with AWS as the default public cloud. The architecture of Metaflow allows for additional public clouds to be supported.

    Adding support for Microsoft Azure might broaden the potential user base, which could increase the adaption rate. This, in turn, could lead to increased community attention.

    enhancement 
    opened by leifericf 21
  • Support for AWS Step functions

    Support for AWS Step functions

    Metaflow on AWS currently requires a human-in-the-loop to execute and cannot automatically be scheduled. Metaflow could be made to work with AWS Step functions to allow the orchestration of Metaflow steps to be done by AWS.

    enhancement 
    opened by romain-intel 20
  • Typo repair and PEP8 cleanup

    Typo repair and PEP8 cleanup

    I've made a number of changes to address misspellings, grammar issues, and other text-things that needed clarified. I was pretty aggressive, so please feel free to reject anything that you don't agree with!

    Some of the changes were made to function names with clear misspellings (i.e. "kubernetes" spelled as "kuberentes")

    ok-to-test 
    opened by jimbudarz 18
  • Support for Kubernetes (with Argo)

    Support for Kubernetes (with Argo)

    Another implementation of #16

    This idea is to provide metaflow with native kubernetes implementation using Argo (https://github.com/argoproj/argo) for the workflow part.

    enhancement 
    opened by nlaille 17
  • Adding support for Azure Blob Storage as a datastore

    Adding support for Azure Blob Storage as a datastore

    The primary change is implementing AzureStorage (analogous to existing S3Storage, LocalStorage). We are consciously deferring the decision of having first class "data tools" support for Azure.

    There are some necessary changes to ensure full Azure support on all Metaflow surfaces:

    • includefile
    • conda
    • cards
    • mflog
    • kubernetes
    • argo

    We take care to ensure there is no cross disruption to users not using Azure. More specifically:

    • Users need to setup AWS dependencies (boto3, config params), iff they are using AWS.
    • Users need to setup Azure dependencies (azure SDK libs, config params), iff they are using Azure.

    We aggressively use local imports to achieve this.

    Much effort was also spent to ensure good performance of Metaflow's usage of Azure Blob Storage. See context docs for more details.

    Some docs for context:

    ok-to-test mergeable 
    opened by jackie-ob 15
  • Parse environment variables passed from CLI

    Parse environment variables passed from CLI

    Right now if one attempted to pass environment variables to the @environment decorator using the following type of syntax:

    python workflow/myflow.py --with environment:vars=FOO:$(FOO),BAR:$(BAR)
    

    vars would incorrectly register as a string rather than parsing the the contents as a dict.

    This PR enables parsing of environment for the environment decorator which are passed using --with

    opened by LarsDu 14
  • Documentation / Explanation on how to use GPU

    Documentation / Explanation on how to use GPU

    Problem

    Not clear how GPU will be used. If Metaflow looks after installing NVIDA drivers or utilise nvidea toolkit doccker instances. If using the G/P/INF instances would automatically enable using GPU.

    In the documentation, I can only see Using AWS Batch where GPU is referenced.

    Note that in this case the resources decorator is used as a prescription for the size of the box that Batch should run the job on; please be sure that this resource requirement can be met. In addition to cpu and memory you can specify gpu=N to request N GPUs for the instance.

    Does this mean regardless with the EC2 instance type, GPU will be allocated in someway e.g. AWS Elastic Inference?

    A Review of Netflix’s Metaflow tells:

    You cannot specify the type of GPU requested with the @resources context manager. AWS offers various types of GPUs, and they can be added to your AWS Batch cluster, but selecting the GPU you would like to use for your machine learning model is not possible in Metaflow. In Metaflow you can use the @resources decorator to define the required resources of a step. These resources will then be provided by AWS Batch if they are available in the cluster. The decorator also allows you to request GPUs

    In the Gitter Metaflow/Community, there are multiple mentions GPU but not clear what are the right steps to use GPU.

    Savin @savingoyal Jan 20 16:58 @amanbedi23 Looks like you need to use a custom GPU enabled AMI when setting up your batch compute environment with G2 instances. For P2 instanced, amazon will automatically provision GPU enabled AMIs.

    Savin @savingoyal Jan 21 15:27 Are you using a custom AMI for your instanced. AWS Batch for p instances launches with the appropriate accelerator enabled AMIs out of the box. Yes, we have run GPU workloads on AWS Batch using metaflow.

    russellbrooks @russellbrooks Jan 23 17:29 @amanbedi23 Your best bet is probably to build your own docker image with most/all the dependencies you expect to encounter, upload that to ECR, and then reference the image in the @batch decorator or when invoking the flow from the CLI (--with batch:image=YourCustomImageName:latest). Another benefit of this approach is better performance at runtime and less copy/pasted conda decorators for commonly shared dependencies.

    Request

    Please update the documentation to detail the requirements, steps on how to use GPU.

    • Do we need to setup a custom AMI or need to use specific ML AMI from AWS unless EC2 instance type is P?
    • Does @resource is enough to use GPU regardless the EC2 instance type?
    • Does Metaflow looks after the NVIDIA driver installation or use Nvidia Toolkit docker by default? How GPU drivers are handled in the docker instances managed by Metarflow?
    opened by oonisim 14
  • Pytorch Parallel Decorator and test

    Pytorch Parallel Decorator and test

    Follow-up for @parallel, with @pytorch_parallel that sets the environment for Pytorch's DDP (distributed data parallel).

    Making this work requires small change to the parallel decorator so that the initilaiization code is run at the right time.

    Added a test flow:

    python  test/parallel/pytorch_parallel_test_flow.py --no-pylint  run
    

    (--no-pylint is needed for torch)

    ok-to-test 
    opened by akyrola 13
  • Task crashed due to CannotInspectContainerError: Could not transition to inspecting; timed out after waiting 30s

    Task crashed due to CannotInspectContainerError: Could not transition to inspecting; timed out after waiting 30s

    Been seeing the following issue in our flows.

    Hypothesis I have explored.

    • Doesn't seem to be any permission or improper setup thing, since dozens of identical branches triggered through foreach process just fine. And they have the exact s3 roles and permissions, so not even sure what the credentials could not be located message indicates.
    • it executes every line of code from the step and does as expected, but then it leads to this timeout error during container exit at the end which makes the entire flow go haywire and fail.
    • I thought maybe it has something to do with trying to snapshot large dataframes, but this error seems to appear randomly regardless of the amount of data being processed in the task.

    Any insights?

    
    timestamp | message
    -- | --
    1.6107E+12 | 2021-01-15 14:47:29.067 [40293/process_a_provider/211754 (pid 17969)]   Task is starting.
    1.6107E+12 | 2021-01-15 14:47:35.168 [40293/process_a_provider/211754 (pid 17969)]   [c94edb3a-091c-4e35-a885-d9bc00ba2bad] Task is starting (status SUBMITTED)...
    1.6107E+12 | 2021-01-15 14:48:01.009 [40293/process_a_provider/211754 (pid 17969)]   [c94edb3a-091c-4e35-a885-d9bc00ba2bad] Task is starting (status RUNNABLE)...
    1.6107E+12 | 2021-01-15 14:48:01.009 [40293/process_a_provider/211754 (pid 17969)]   [c94edb3a-091c-4e35-a885-d9bc00ba2bad] Task is starting (status RUNNABLE)...
    1.6107E+12 | 2021-01-15 14:48:31.192 [40293/process_a_provider/211754 (pid 17969)]   [c94edb3a-091c-4e35-a885-d9bc00ba2bad] Task is starting (status RUNNABLE)...
    1.6107E+12 | 2021-01-15 14:49:01.335 [40293/process_a_provider/211754 (pid 17969)]   [c94edb3a-091c-4e35-a885-d9bc00ba2bad] Task is starting (status RUNNABLE)...
    1.6107E+12 | 2021-01-15 14:49:31.467 [40293/process_a_provider/211754 (pid 17969)]   [c94edb3a-091c-4e35-a885-d9bc00ba2bad] Task is starting (status RUNNABLE)...
    1.6107E+12 | 2021-01-15 14:50:01.628 [40293/process_a_provider/211754 (pid 17969)]   [c94edb3a-091c-4e35-a885-d9bc00ba2bad] Task is starting (status RUNNABLE)...
    1.6107E+12 | 2021-01-15 14:50:33.037 [40293/process_a_provider/211754 (pid 17969)]   [c94edb3a-091c-4e35-a885-d9bc00ba2bad] Task is starting (status RUNNABLE)...
    1.6107E+12 | 2021-01-15 14:50:35.746 [40293/process_a_provider/211754 (pid 17969)]   [c94edb3a-091c-4e35-a885-d9bc00ba2bad] Task is starting (status STARTING)...
    1.6107E+12 | 2021-01-15 14:51:05.856 [40293/process_a_provider/211754 (pid 17969)]   [c94edb3a-091c-4e35-a885-d9bc00ba2bad] Task is starting (status STARTING)...
    1.6107E+12 | 2021-01-15 14:51:29.848 [40293/process_a_provider/211754 (pid 17969)]   [c94edb3a-091c-4e35-a885-d9bc00ba2bad] Task is starting (status RUNNING)...
    1.6107E+12 | 2021-01-15 14:51:39.622 [40293/process_a_provider/211754 (pid 17969)]   [c94edb3a-091c-4e35-a885-d9bc00ba2bad] Setting up task environment.
    1.6107E+12 | 2021-01-15 14:59:17.185 [40293/process_a_provider/211754 (pid 17969)]   [c94edb3a-091c-4e35-a885-d9bc00ba2bad] WARNING: Retrying (Retry(total=4,   connect=None, read=None, redirect=None, status=None)) after connection broken   by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]   certificate verify failed: unable to get local issuer certificate   (_ssl.c:1123)'))': /simple/awscli/
    1.6107E+12 | 2021-01-15 15:03:31.869 [40293/process_a_provider/211754 (pid 17969)]   [c94edb3a-091c-4e35-a885-d9bc00ba2bad] Downloading code package.
    1.6107E+12 | 2021-01-15 15:03:41.894 [40293/process_a_provider/211754 (pid 17969)]   [c94edb3a-091c-4e35-a885-d9bc00ba2bad] Code package downloaded.
    1.6107E+12 | 2021-01-15 15:03:47.287 [40293/process_a_provider/211754 (pid 17969)]   [c94edb3a-091c-4e35-a885-d9bc00ba2bad] Bootstrapping environment.
    1.6107E+12 | 2021-01-15 15:08:27.341 [40293/process_a_provider/211754 (pid 17969)]   [c94edb3a-091c-4e35-a885-d9bc00ba2bad] Environment bootstrapped.
    1.6107E+12 | 2021-01-15 15:09:32.012 [40293/process_a_provider/211754 (pid 17969)]   [c94edb3a-091c-4e35-a885-d9bc00ba2bad] Task is starting.
    1.6107E+12 | 2021-01-15 15:10:31.520 [40293/process_a_provider/211754 (pid 17969)]   [c94edb3a-091c-4e35-a885-d9bc00ba2bad] Provider processed successfully -   0655011
    1.6107E+12 | 2021-01-15 15:11:05.938 [40293/process_a_provider/211754 (pid 17969)]   Batch error:
    1.6107E+12 | 2021-01-15 15:11:05.938 [40293/process_a_provider/211754 (pid 17969)]   Task crashed due to CannotInspectContainerError: Could not transition to   inspecting; timed out after waiting 30s .This could be a transient error. Use   @retry to retry.
    1.6107E+12 | 2021-01-15 15:11:05.938 [40293/process_a_provider/211754 (pid   17969)]
    1.6107E+12 | 2021-01-15 15:11:06.448 [40293/process_a_provider/211754 (pid 17969)]   Task failed.
    1.6107E+12 | 2021-01-15 15:11:07.060 [40293/process_a_provider/211754 (pid 18823)]   Task fallback is starting to handle the failure.
    1.6107E+12 | 2021-01-15 15:11:07.775 [40293/process_a_provider/211754 (pid 18823)] S3   datastore operation _put_s3_object failed (Unable to locate credentials).   Retrying 7 more times..
    1.6107E+12 | 2021-01-15 15:11:09.790 [40293/process_a_provider/211754 (pid 18823)] S3   datastore operation _put_s3_object failed (Unable to locate credentials).   Retrying 6 more times..
    1.6107E+12 | 2021-01-15 15:11:16.810 [40293/process_a_provider/211754 (pid 18823)] S3   datastore operation _put_s3_object failed (Unable to locate credentials).   Retrying 5 more times..
    1.6107E+12 | 2021-01-15 15:11:20.825 [40293/process_a_provider/211754 (pid 18823)] S3   datastore operation _put_s3_object failed (Unable to locate credentials).   Retrying 4 more times..
    1.6107E+12 | 2021-01-15 15:11:32.850 [40293/process_a_provider/211754 (pid 18823)] S3   datastore operation _put_s3_object failed (Unable to locate credentials).   Retrying 3 more times..
    1.6107E+12 | 2021-01-15 15:11:50.880 [40293/process_a_provider/211754 (pid 18823)] S3   datastore operation _put_s3_object failed (Unable to locate credentials).   Retrying 2 more times..
    1.6107E+12 | 2021-01-15 15:12:27.929 [40293/process_a_provider/211754 (pid 18823)] S3   datastore operation _put_s3_object failed (Unable to locate credentials).   Retrying 1 more times..
    1.6107E+12 | 2021-01-15 15:13:31.995 [40293/process_a_provider/211754 (pid 18823)]   Internal error
    1.6107E+12 | 2021-01-15 15:13:31.999 [40293/process_a_provider/211754 (pid 18823)]   Traceback (most recent call last):
    1.6107E+12 | 2021-01-15 15:13:32.000 [40293/process_a_provider/211754 (pid 18823)]   File "/tmp/tmpolxsu4uf/metaflow/cli.py", line 853, in main
    1.6107E+12 | 2021-01-15 15:13:32.000 [40293/process_a_provider/211754 (pid 18823)]   start(auto_envvar_prefix='METAFLOW', obj=state)
    1.6107E+12 | 2021-01-15 15:13:32.000 [40293/process_a_provider/211754 (pid 18823)]   File   "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/click/core.py",   line 764, in __call__
    1.6107E+12 | 2021-01-15 15:13:32.000 [40293/process_a_provider/211754 (pid 18823)]   return self.main(args, kwargs)
    1.6107E+12 | 2021-01-15 15:13:32.000 [40293/process_a_provider/211754 (pid 18823)]   File   "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/click/core.py",   line 717, in main
    1.6107E+12 | 2021-01-15 15:13:32.000 [40293/process_a_provider/211754 (pid 18823)] rv   = self.invoke(ctx)
    1.6107E+12 | 2021-01-15 15:13:32.000 [40293/process_a_provider/211754 (pid 18823)]   File   "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/click/core.py",   line 1137, in invoke
    1.6107E+12 | 2021-01-15 15:13:32.000 [40293/process_a_provider/211754 (pid 18823)]   return _process_result(sub_ctx.command.invoke(sub_ctx))
    1.6107E+12 | 2021-01-15 15:13:32.000 [40293/process_a_provider/211754 (pid 18823)]   File   "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/click/core.py",   line 956, in invoke
    1.6107E+12 | 2021-01-15 15:13:32.000 [40293/process_a_provider/211754 (pid 18823)]   return ctx.invoke(self.callback, ctx.params)
    1.6107E+12 | 2021-01-15 15:13:32.000 [40293/process_a_provider/211754 (pid 18823)]   File   "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/click/core.py",   line 555, in invoke
    1.6107E+12 | 2021-01-15 15:13:32.001 [40293/process_a_provider/211754 (pid 18823)]   return callback(args, kwargs)
    1.6107E+12 | 2021-01-15 15:13:32.001 [40293/process_a_provider/211754 (pid 18823)]   File   "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/click/decorators.py",   line 27, in new_func
    1.6107E+12 | 2021-01-15 15:13:32.001 [40293/process_a_provider/211754 (pid 18823)]   return f(get_current_context().obj, args, kwargs)
    1.6107E+12 | 2021-01-15 15:13:32.001 [40293/process_a_provider/211754 (pid 18823)]   File "/tmp/tmpolxsu4uf/metaflow/cli.py", line 430, in step
    1.6107E+12 | 2021-01-15 15:13:32.001 [40293/process_a_provider/211754 (pid 18823)]   max_user_code_retries)
    1.6107E+12 | 2021-01-15 15:13:32.001 [40293/process_a_provider/211754 (pid 18823)]   File "/tmp/tmpolxsu4uf/metaflow/task.py", line 278, in run_step
    1.6107E+12 | 2021-01-15 15:13:32.001 [40293/process_a_provider/211754 (pid 18823)]   monitor=self.monitor)
    1.6107E+12 | 2021-01-15 15:13:32.001 [40293/process_a_provider/211754 (pid 18823)]   File "/tmp/tmpolxsu4uf/metaflow/datastore/s3.py", line 34, in   __init__
    1.6107E+12 | 2021-01-15 15:13:32.001 [40293/process_a_provider/211754 (pid 18823)]   super(S3DataStore, self).__init__(args, kwargs)
    1.6107E+12 | 2021-01-15 15:13:32.001 [40293/process_a_provider/211754 (pid 18823)]   File "/tmp/tmpolxsu4uf/metaflow/datastore/datastore.py", line 362,   in __init__
    1.6107E+12 | 2021-01-15 15:13:32.002 [40293/process_a_provider/211754 (pid 18823)]   self.save_metadata('attempt', {'time': time.time()})
    1.6107E+12 | 2021-01-15 15:13:32.002 [40293/process_a_provider/211754 (pid 18823)]   File "/tmp/tmpolxsu4uf/metaflow/datastore/datastore.py", line 49,   in method
    1.6107E+12 | 2021-01-15 15:13:32.002 [40293/process_a_provider/211754 (pid 18823)]   return f(self, args, kwargs)
    1.6107E+12 | 2021-01-15 15:13:32.002 [40293/process_a_provider/211754 (pid 18823)]   File "/tmp/tmpolxsu4uf/metaflow/datastore/s3.py", line 168, in   save_metadata
    1.6107E+12 | 2021-01-15 15:13:32.002 [40293/process_a_provider/211754 (pid 18823)]   self._put_s3_object(path, json.dumps(data).encode('utf-8'))
    1.6107E+12 | 2021-01-15 15:13:32.002 [40293/process_a_provider/211754 (pid 18823)]   File "/tmp/tmpolxsu4uf/metaflow/datastore/util/s3util.py", line 37,   in retry_wrapper
    1.6107E+12 | 2021-01-15 15:13:32.002 [40293/process_a_provider/211754 (pid 18823)]   raise last_exc
    1.6107E+12 | 2021-01-15 15:13:32.002 [40293/process_a_provider/211754 (pid 18823)]   File "/tmp/tmpolxsu4uf/metaflow/datastore/util/s3util.py", line 21,   in retry_wrapper
    1.6107E+12 | 2021-01-15 15:13:32.002 [40293/process_a_provider/211754 (pid 18823)]   return f(self, args, kwargs)
    1.6107E+12 | 2021-01-15 15:13:32.002 [40293/process_a_provider/211754 (pid 18823)]   File "/tmp/tmpolxsu4uf/metaflow/datastore/s3.py", line 65, in   _put_s3_object
    1.6107E+12 | 2021-01-15 15:13:32.002 [40293/process_a_provider/211754 (pid 18823)]   self.s3.upload_fileobj(buf, url.netloc, url.path.lstrip('/'))
    1.6107E+12 | 2021-01-15 15:13:32.002 [40293/process_a_provider/211754 (pid 18823)]   File   "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/boto3/s3/inject.py",   line 539, in upload_fileobj
    1.6107E+12 | 2021-01-15 15:13:32.003 [40293/process_a_provider/211754 (pid 18823)]   return future.result()
    1.6107E+12 | 2021-01-15 15:13:32.003 [40293/process_a_provider/211754 (pid 18823)]   File   "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/s3transfer/futures.py",   line 106, in result
    1.6107E+12 | 2021-01-15 15:13:32.003 [40293/process_a_provider/211754 (pid 18823)]   return self._coordinator.result()
    1.6107E+12 | 2021-01-15 15:13:32.003 [40293/process_a_provider/211754 (pid 18823)]   File   "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/s3transfer/futures.py",   line 265, in result
    1.6107E+12 | 2021-01-15 15:13:32.003 [40293/process_a_provider/211754 (pid 18823)]   raise self._exception
    1.6107E+12 | 2021-01-15 15:13:32.003 [40293/process_a_provider/211754 (pid 18823)]   File   "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/s3transfer/tasks.py",   line 126, in __call__
    1.6107E+12 | 2021-01-15 15:13:32.003 [40293/process_a_provider/211754 (pid 18823)]   return self._execute_main(kwargs)
    1.6107E+12 | 2021-01-15 15:13:32.003 [40293/process_a_provider/211754 (pid 18823)]   File   "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/s3transfer/tasks.py",   line 150, in _execute_main
    1.6107E+12 | 2021-01-15 15:13:32.003 [40293/process_a_provider/211754 (pid 18823)]   return_value = self._main(kwargs)
    1.6107E+12 | 2021-01-15 15:13:32.003 [40293/process_a_provider/211754 (pid 18823)]   File   "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/s3transfer/upload.py",   line 692, in _main
    1.6107E+12 | 2021-01-15 15:13:32.003 [40293/process_a_provider/211754 (pid 18823)]   client.put_object(Bucket=bucket, Key=key, Body=body, extra_args)
    1.6107E+12 | 2021-01-15 15:13:32.004 [40293/process_a_provider/211754 (pid 18823)]   File   "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/botocore/client.py",   line 357, in _api_call
    1.6107E+12 | 2021-01-15 15:13:32.004 [40293/process_a_provider/211754 (pid 18823)]   return self._make_api_call(operation_name, kwargs)
    1.6107E+12 | 2021-01-15 15:13:32.004 [40293/process_a_provider/211754 (pid 18823)]   File   "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/botocore/client.py",   line 648, in _make_api_call
    1.6107E+12 | 2021-01-15 15:13:32.111 [40293/process_a_provider/211754 (pid 18823)]   operation_model, request_dict, request_context)
    1.6107E+12 | 2021-01-15 15:13:32.111 [40293/process_a_provider/211754 (pid 18823)]   File   "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/botocore/client.py",   line 667, in _make_request
    1.6107E+12 | 2021-01-15 15:13:32.111 [40293/process_a_provider/211754 (pid 18823)]   return self._endpoint.make_request(operation_model, request_dict)
    1.6107E+12 | 2021-01-15 15:13:32.111 [40293/process_a_provider/211754 (pid 18823)]   File   "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/botocore/endpoint.py",   line 102, in make_request
    1.6107E+12 | 2021-01-15 15:13:32.111 [40293/process_a_provider/211754 (pid 18823)]   return self._send_request(request_dict, operation_model)
    1.6107E+12 | 2021-01-15 15:13:32.111 [40293/process_a_provider/211754 (pid 18823)]   File   "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/botocore/endpoint.py",   line 132, in _send_request
    1.6107E+12 | 2021-01-15 15:13:32.111 [40293/process_a_provider/211754 (pid 18823)]   request = self.create_request(request_dict, operation_model)
    1.6107E+12 | 2021-01-15 15:13:32.111 [40293/process_a_provider/211754 (pid 18823)]   File   "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/botocore/endpoint.py",   line 116, in create_request
    1.6107E+12 | 2021-01-15 15:13:32.111 [40293/process_a_provider/211754 (pid 18823)]   operation_name=operation_model.name)
    1.6107E+12 | 2021-01-15 15:13:32.111 [40293/process_a_provider/211754 (pid 18823)]   File   "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/botocore/hooks.py",   line 356, in emit
    1.6107E+12 | 2021-01-15 15:13:32.111 [40293/process_a_provider/211754 (pid 18823)]   return self._emitter.emit(aliased_event_name, kwargs)
    1.6107E+12 | 2021-01-15 15:13:32.111 [40293/process_a_provider/211754 (pid 18823)]   File   "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/botocore/hooks.py",   line 228, in emit
    1.6107E+12 | 2021-01-15 15:13:32.112 [40293/process_a_provider/211754 (pid 18823)]   return self._emit(event_name, kwargs)
    1.6107E+12 | 2021-01-15 15:13:32.112 [40293/process_a_provider/211754 (pid 18823)]   File   "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/botocore/hooks.py",   line 211, in _emit
    1.6107E+12 | 2021-01-15 15:13:32.112 [40293/process_a_provider/211754 (pid 18823)]   response = handler(kwargs)
    1.6107E+12 | 2021-01-15 15:13:32.112 [40293/process_a_provider/211754 (pid 18823)]   File   "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/botocore/signers.py",   line 90, in handler
    1.6107E+12 | 2021-01-15 15:13:32.112 [40293/process_a_provider/211754 (pid 18823)]   return self.sign(operation_name, request)
    1.6107E+12 | 2021-01-15 15:13:32.112 [40293/process_a_provider/211754 (pid 18823)]   File   "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/botocore/signers.py",   line 157, in sign
    1.6107E+12 | 2021-01-15 15:13:32.112 [40293/process_a_provider/211754 (pid 18823)]   auth.add_auth(request)
    1.6107E+12 | 2021-01-15 15:13:32.112 [40293/process_a_provider/211754 (pid 18823)]   File   "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/botocore/auth.py",   line 425, in add_auth
    1.6107E+12 | 2021-01-15 15:13:32.112 [40293/process_a_provider/211754 (pid 18823)]   super(S3SigV4Auth, self).add_auth(request)
    1.6107E+12 | 2021-01-15 15:13:32.112 [40293/process_a_provider/211754 (pid 18823)]   File   "/home/ec2-user/setup-workspace/miniconda/envs/metaflow_AnalysisFlow_linux-64_4b7b525aec038862f563ddf6c526fef7d88b1900/lib/python3.6/site-packages/botocore/auth.py",   line 357, in add_auth
    1.6107E+12 | 2021-01-15 15:13:32.112 [40293/process_a_provider/211754 (pid 18823)]   raise NoCredentialsError
    1.6107E+12 | 2021-01-15 15:13:32.112 [40293/process_a_provider/211754 (pid 18823)]   botocore.exceptions.NoCredentialsError: Unable to locate credentials
    1.6107E+12 | 2021-01-15 15:13:32.112 [40293/process_a_provider/211754 (pid   18823)]
    1.6107E+12 | 2021-01-15 15:13:32.477 [40293/process_a_provider/211754 (pid 18823)]   Task failed.
    1.6107E+12 | 2021-01-15 15:13:33.071 [40293/process_a_provider/211754 (pid 18937)]   Task fallback is starting to handle the failure.
    1.6107E+12 | 2021-01-15 15:13:33.763 [40293/process_a_provider/211754 (pid 18937)] S3   datastore operation _put_s3_object failed (Unable to locate credentials).   Retrying 7 more times..
    1.6107E+12 | 2021-01-15 15:13:39.179 [40293/process_a_provider/211754 (pid 18937)] S3   datastore operation _put_s3_object failed (Unable to locate credentials).   Retrying 6 more times..
    1.6107E+12 | 2021-01-15 15:13:42.798 [40293/process_a_provider/211754 (pid 18937)] S3   datastore operation _put_s3_object failed (Unable to locate credentials).   Retrying 5 more times..
    1.6107E+12 | 2021-01-15 15:13:49.822 [40293/process_a_provider/211754 (pid 18937)] S3   datastore operation _put_s3_object failed (Unable to locate credentials).   Retrying 4 more times..
    1.6107E+12 | 2021-01-15 15:14:00.841 [40293/process_a_provider/211754 (pid 18937)] S3   datastore operation _put_s3_object failed (Unable to locate credentials).   Retrying 3 more times..
    1.6107E+12 | 2021-01-15 15:14:18.866 [40293/process_a_provider/211754 (pid 18937)] S3   datastore operation _put_s3_object failed (Unable to locate credentials).   Retrying 2 more times..
    
    
    
    opened by Viveckh 13
  • Conditional branch documentation + example usage

    Conditional branch documentation + example usage

    From the flowspec docstring:

    - Conditional branch:
        self.next(self.if_true, self.if_false, condition='boolean_variable')
      In this situation, both `if_true` and `if_false` are methods in the current class
      decorated with the `@step` decorator and `boolean_variable` is a variable name
      in the current class that evaluates to True or False. The `if_true` step will be
      executed if thecondition variable evaluates to True and the `if_false` step will
      be executed otherwise
    

    It'd be great to have this mentioned on Metaflow's website with an example.

    On a related note, this capability makes it easier to introduce cycles into the DAG and while the documentation mentions:

    Metaflow infers a directed (typically acyclic) graph based on the transitions between step functions.

    The acyclicity seems to be checked by the linter, however that piece of validation is not disabled using the --no-pylint flag. It's alluded to the possibility of a graph with cycles, but it doesn't seem possible to do for now.

    Example:

    from metaflow import FlowSpec, step
    
    
    class TestFlowConditional(FlowSpec):
        """
        A toy flow to mimic a hyperparameter tuning strategy.
        
        The flow performs the following steps:
        1) Load data.
        2) Generate hyperparameter candidates.
        3) Fan-out training over hyperparameter candidates to evaluate using foreach.
        4) Join results.
        5) Conditionally stop at max iterations or keep evaluating.
        """
    
        @step
        def start(self):
            # self.data = ...
            self._iteration = 0
            self._max_iteration = 15
            self._num_candidates = 3
            self.results = []
            self.next(self.generate_candidates)
    
        @step
        def generate_candidates(self):
            candidates = []
            for _ in range(self._num_candidates):
                candidate = {
                    "hyperparameters": { 
                        # ... 
                    },
                }
                candidates.append(candidate)
            self.candidates = candidates
            self._iteration += len(candidates)
            self.next(self.train, foreach='candidates')
    
        @step
        def train(self):
            hyperparams = self.input['hyperparameters']
            
            # ...
            
            self.next(self.join)
    
        @step
        def join(self, inputs):
            """
            Combine results for hyperparameter candidates.
            """
            # ...
            self.next(self.should_stop)
    
        @step
        def should_stop(self):
            """
            Conditional branch to end when max iterations is reached, otherwise evaluate more candidates.
            """
            self._done = self._iteration < self._max_iteration
            self.next(self.end, self.generate_candidates, condition='_done')
    
        @step
        def end(self):
            print("Finished.")
    
    if __name__ == '__main__':
        TestFlowConditional()
    
    python TestMetaflowConditional.py --no-pylint run
    

    Results in:

    Metaflow 2.0.1 executing TestFlowConditional for user:russell
    Validating your flow...
        Validity checker found an issue on line 62:
        There is a loop in your flow: generate_candidates->train->join->should_stop->generate_candidates. Break the loop by fixing self.next() transitions.
    

    Keep up the great work and I've been enjoying Metaflow so far!

    opened by russellbrooks 13
  • Add support for custom tags at run time

    Add support for custom tags at run time

    It would be nice one could add custom tags via a flow/step decorator or environmental variables. Currently the tagging system seems optimized for post-run tagging, but often times one wants to tag a specific run in order to find that run later. I would propose either a decorator on the flow or environmental variables.

    # within flow decorator or on a step
    @project(name="my_project", tags=("tag1", "tag2:tag2value" , "tag3"))
    class MyFlow(FlowSpec):
      @tags(("start_tag1", "start_tag2:start_tag2_value"))
      def start(self):
        ...
    
    # with an env var
    export METAFLOW_USER_TAGS="tag1|tag2:tag2value|tag3"
    export METAFLOW_USER_TAGS_DELIMITER="|"
    
    opened by dhpollack 0
  • Introduce support for micromamba for @conda

    Introduce support for micromamba for @conda

    Includes -

    1. Integrating with micromamba server https://github.com/mamba-org/mamba/pull/2185 for env resolution and set up
    2. Pipelined caching of conda packages
    3. Support for virtual packages
    4. Utilizing micromamba for cached env setup on remote environments
    opened by savingoyal 0
  • Invalidate .metaflow folder if suitable changes to metaflow config are detected

    Invalidate .metaflow folder if suitable changes to metaflow config are detected

    If certain changes are made to metaflow config (or AWS config) then the state in .metaflow folder may not be reflective of the new reality (latest run ids, conda lock files, local metadata etc). In those scenarios, we can invalidate the folder rather that asking the user to nuke it manually.

    opened by savingoyal 0
  • Record GIT information in metadata

    Record GIT information in metadata

    It may be interesting to record the commit and possibly the current diff file (similar to what Comet does) when running a flow to allow the user to recreate the development environment (and not just the code package).

    opened by romain-intel 0
  • Support AWS inferentia instances (e.g. `inf1.xlarge`, `trn1.2xlarge`)

    Support AWS inferentia instances (e.g. `inf1.xlarge`, `trn1.2xlarge`)

    AWS Inferentia (and Tranium) instances are a custom ASICs designed specifically for running ML models.

    We'd like to run inferentia based workflows on AWS Batch using Metaflow. This would support workflows with Metaflow such as:

    • Using neuron for training
    • Compiling trained models with neuron
    • Benchmarking neuron models
    • Running neuron inference

    The necessary changes are (using the API definitions here and docs for Inferentia on AWS Batch here):

    • Support adding AWS::Batch::JobDefinition.ContainerProperties.LinuxParameters.Devices in the batch job definition to mount the neuron devices.

      For example:

      	"devices": [
      	    {
      	        "containerPath": "/dev/neuron0",
      	        "hostPath": "/dev/neuron0",
      	        "permissions": [
      	            "read",
      	            "write"
      	        ]
      	    }
      	]
      
    • Support setting AWS::Batch::JobDefinition.Priviliged to true - this is in lieu of support for setting capabilities on AWS Batch Job Definitions (https://github.com/aws/containers-roadmap/issues/1916) which is more specific and preferable

      For example:

      	"privileged": true
      
    opened by Limess 0
Releases(2.7.18)
  • 2.7.18(Dec 8, 2022)

    What's Changed

    • Adds check for tutorials dir and flattens if necessary by @ashrielbrian in https://github.com/Netflix/metaflow/pull/1211
    • Fix bug with datastore backend instantiation by @savingoyal in https://github.com/Netflix/metaflow/pull/1210

    New Contributors

    • @ashrielbrian made their first contribution in https://github.com/Netflix/metaflow/pull/1211

    Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.17...2.7.18

    Source code(tar.gz)
    Source code(zip)
  • 2.7.17(Dec 7, 2022)

    What's Changed

    • Fix regression causing CL tool to not work. by @romain-intel in https://github.com/Netflix/metaflow/pull/1209
    • Bump qs from 6.5.2 to 6.5.3 in /metaflow/plugins/cards/ui by @dependabot in https://github.com/Netflix/metaflow/pull/1208

    Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.16...2.7.17

    Source code(tar.gz)
    Source code(zip)
  • 2.7.16(Dec 6, 2022)

    What's Changed

    • Deal with transient errors (like SlowDowns) more effectively for S3 by @romain-intel in https://github.com/Netflix/metaflow/pull/1186
    • Fix/move data files by @romain-intel in https://github.com/Netflix/metaflow/pull/1206

    Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.15...2.7.16

    Source code(tar.gz)
    Source code(zip)
  • 2.7.15(Dec 2, 2022)

    What's Changed

    • Handle aborted Kubernetes workloads. by @shrinandj in https://github.com/Netflix/metaflow/pull/1195
    • Bump loader-utils from 3.2.0 to 3.2.1 in /metaflow/plugins/cards/ui by @dependabot in https://github.com/Netflix/metaflow/pull/1194
    • Fix ._orig access for submodules for MF extensions by @romain-intel in https://github.com/Netflix/metaflow/pull/1174
    • Update black to latest version by @savingoyal in https://github.com/Netflix/metaflow/pull/1199
    • allow equal sign in decorator spec values by @amerberg in https://github.com/Netflix/metaflow/pull/1197
    • Typo repair and PEP8 cleanup by @jimbudarz in https://github.com/Netflix/metaflow/pull/1190
    • Pin GH tests to Ubuntu 20.04 by @savingoyal in https://github.com/Netflix/metaflow/pull/1201
    • Set gpu resources correctly "--with kubernetes" by @shrinandj in https://github.com/Netflix/metaflow/pull/1202
    • Clean up configuration variables by @romain-intel in https://github.com/Netflix/metaflow/pull/1183
    • GCP datastore implementation by @jackie-ob in https://github.com/Netflix/metaflow/pull/1135
    • Bump version; remove R tests by @romain-intel in https://github.com/Netflix/metaflow/pull/1204

    New Contributors

    • @shrinandj made their first contribution in https://github.com/Netflix/metaflow/pull/1195
    • @amerberg made their first contribution in https://github.com/Netflix/metaflow/pull/1197
    • @jimbudarz made their first contribution in https://github.com/Netflix/metaflow/pull/1190

    Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.14...2.7.15

    Source code(tar.gz)
    Source code(zip)
  • 2.7.14(Nov 3, 2022)

    What's Changed

    • fix pandas call bug by @mbalajew in https://github.com/Netflix/metaflow/pull/1173
    • Metaflow pathspec in Airflow UI by @valayDave in https://github.com/Netflix/metaflow/pull/1119
    • Allow the input paths to be passed via a file by @romain-intel in https://github.com/Netflix/metaflow/pull/1181
    • Check compatibility for R 4.2 by @savingoyal in https://github.com/Netflix/metaflow/pull/1160
    • issue 1040 fix: apply _sanitize to template names in Argo workflows by @johnaparker in https://github.com/Netflix/metaflow/pull/1180

    New Contributors

    • @mbalajew made their first contribution in https://github.com/Netflix/metaflow/pull/1173
    • @johnaparker made their first contribution in https://github.com/Netflix/metaflow/pull/1180

    Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.13...2.7.14

    Source code(tar.gz)
    Source code(zip)
  • 2.7.13(Oct 18, 2022)

    What's Changed

    • Add cmd extension point to allow MF extensions to extend it by @romain-intel in https://github.com/Netflix/metaflow/pull/1143
    • Fix periodic messages printed at runtime by @romain-intel and @jackie-ob in https://github.com/Netflix/metaflow/pull/1061, https://github.com/Netflix/metaflow/pull/1151 and https://github.com/Netflix/metaflow/pull/1159
    • Pass datastore_type to validate_environment by @romain-intel in https://github.com/Netflix/metaflow/pull/1152
    • Support kubernetes_conn_id in Airflow integration by @valayDave in https://github.com/Netflix/metaflow/pull/1153
    • Use json to dump/load decorator specs by @romain-intel in https://github.com/Netflix/metaflow/pull/1144
    • argo use kubernetes client class by @oavdeev in https://github.com/Netflix/metaflow/pull/1163
    • Rewrite IncludeFile implementation by @romain-intel in https://github.com/Netflix/metaflow/pull/1109
    • Add options to make card generation faster in some cases by @romain-intel in https://github.com/Netflix/metaflow/pull/1167
    • Env escape improvements and bug fixes by @romain-intel in https://github.com/Netflix/metaflow/pull/1166
    • Allow figures in Image.from_matplotlib by @valayDave in https://github.com/Netflix/metaflow/pull/1147
    • Bump for release by @romain-intel in https://github.com/Netflix/metaflow/pull/1168

    Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.12...2.7.13

    Source code(tar.gz)
    Source code(zip)
  • 2.7.12(Sep 26, 2022)

    The Metaflow 2.7.12 release is a minor release

    What's Changed

    • Make plugins.airflow.plumbing a well-formed module by @savingoyal in https://github.com/Netflix/metaflow/pull/1148

    Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.11...2.7.12

    Source code(tar.gz)
    Source code(zip)
  • 2.7.11(Sep 16, 2022)

    The Metaflow 2.7.11 release is a minor release

    Fixes

    • Fix DeprecationWarning on invalid escape sequence by @tommybrecher in https://github.com/Netflix/metaflow/pull/1133
    • fix cpu value formatting for aws batch/sfn by @oavdeev in https://github.com/Netflix/metaflow/pull/1140

    New Contributors

    • @tommybrecher made their first contribution in https://github.com/Netflix/metaflow/pull/1133

    Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.10...2.7.11

    Source code(tar.gz)
    Source code(zip)
  • 2.7.10(Sep 8, 2022)

    What's Changed

    • Card bug fix when task-ids are non-unique by @valayDave in https://github.com/Netflix/metaflow/pull/1126
    • Bump version to 2.7.10 to prepare for release by @romain-intel in https://github.com/Netflix/metaflow/pull/1136

    Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.9...2.7.10

    Source code(tar.gz)
    Source code(zip)
  • 2.7.9(Sep 5, 2022)

    What's Changed

    • Fix issue with S3 URLs for packages by @savingoyal in https://github.com/Netflix/metaflow/pull/1130

    Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.8...2.7.9

    Source code(tar.gz)
    Source code(zip)
  • 2.7.8(Sep 3, 2022)

    What's Changed

    • Support airflow with metaflow on azure by @valayDave in https://github.com/Netflix/metaflow/pull/1127
    • Fix issue with S3 invocation for conda bootstrap by @savingoyal in https://github.com/Netflix/metaflow/pull/1128

    Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.7...2.7.8

    Source code(tar.gz)
    Source code(zip)
  • 2.7.7(Aug 25, 2022)

    Metaflow 2.7.7 Release Notes

    The Metaflow 2.7.7 release is a minor release

    Fixes:

    • Fix an issue with get_cards not respecting a Task's ds-root in https://github.com/Netflix/metaflow/pull/1111
    • more robust resource type conversions for aws batch/sfn in https://github.com/Netflix/metaflow/pull/1118

    Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.6...2.7.7

    Source code(tar.gz)
    Source code(zip)
  • 2.7.6(Aug 9, 2022)

    What's Changed

    • Fix another issue with the escape hatch and paths by @romain-intel in https://github.com/Netflix/metaflow/pull/1105
    • Bump to 2.7.6 by @romain-intel in https://github.com/Netflix/metaflow/pull/1106

    Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.5...2.7.6

    Source code(tar.gz)
    Source code(zip)
  • 2.7.5(Aug 5, 2022)

    What's Changed

    • Fix for env_escape bug when importing local packages by @hunsdiecker in https://github.com/Netflix/metaflow/pull/1100
    • Bump to 2.7.5 by @romain-intel in https://github.com/Netflix/metaflow/pull/1102

    New Contributors

    • @hunsdiecker made their first contribution in https://github.com/Netflix/metaflow/pull/1100

    Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.4...2.7.5

    Source code(tar.gz)
    Source code(zip)
  • 2.7.4(Aug 3, 2022)

    What's Changed

    • Fix docstrings for the upcoming API reference (no functional changes!) by @tuulos in https://github.com/Netflix/metaflow/pull/1076
    • Move a sys.path modification in s3op to main by @romain-intel in https://github.com/Netflix/metaflow/pull/1095
    • Airflow Support by @valayDave in https://github.com/Netflix/metaflow/pull/1094
    • Move sys.path insert earlier in s3op.py by @romain-intel in https://github.com/Netflix/metaflow/pull/1098
    • bump version to 2.7.4 for release by @savingoyal in https://github.com/Netflix/metaflow/pull/1099

    Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.3...2.7.4

    Source code(tar.gz)
    Source code(zip)
  • 2.7.3(Jul 29, 2022)

    Metaflow 2.7.3 Release Notes

    The Metaflow 2.7.3 release is a minor release

    Fixes

    • Fix fractional resoure handling for batch in https://github.com/Netflix/metaflow/pull/1089

    Improvements

    • Metadata version check flag by @mrfalconer in https://github.com/Netflix/metaflow/pull/1088

    Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.2...2.7.3

    Source code(tar.gz)
    Source code(zip)
  • 2.7.2(Jul 14, 2022)

    Metaflow 2.7.2 Release Notes

    Metaflow 2.7.2 is a minor release

    Features

    • Support M1 Macs for @conda in https://github.com/Netflix/metaflow/pull/1077

    Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.1...2.7.2

    Source code(tar.gz)
    Source code(zip)
  • 2.7.1(Jun 17, 2022)

    This is a patch release addressing a behavior of the environment escape mechanism.

    Bug Fixes

    • Previously, if the environment escape mechanism provided a package, a failure would occur if that package was also present in the inner environment. This is now changed and, in that case, the package present is used and the environment escape mechanism is not used.

    Full Changelog: https://github.com/Netflix/metaflow/compare/2.7.0...2.7.1

    Source code(tar.gz)
    Source code(zip)
  • 2.7.0(Jun 16, 2022)

    This is a minor release which primarily adds the ability to do runtime tagging.

    Features

    • Adds the ability to mutate a run's tags during or after a run. A CLI tool is provided (tag) as well as methods in the client add_tags, replace_tags and remove_tags. If using the Metaflow Metadata service, a version greater than 2.3.0 is required to use this feature.

    Full Changelog: https://github.com/Netflix/metaflow/compare/2.6.3...2.7.0

    Source code(tar.gz)
    Source code(zip)
  • 2.6.3(May 26, 2022)

    Metaflow 2.6.3 Release Notes

    The Metaflow 2.6.3 release is a minor release

    Bug Fixes

    • Fix instance metadata calls for IMDSV2 in https://github.com/Netflix/metaflow/pull/1053

    Full Changelog: https://github.com/Netflix/metaflow/compare/2.6.2...2.6.3

    Source code(tar.gz)
    Source code(zip)
  • 2.6.2(May 26, 2022)

    Metaflow 2.6.2 Release Notes

    The Metaflow 2.6.2 release is a minor release

    Features

    • Support setting default secrets for @kubernetes (#1048 ). Metaflow allows you mount secrets in Kubernetes containers created by tasks. Now you can specify a set of secrets to be mounted by default via METAFLOW_KUBERNETES_SECRETS configuration option, in addition to existing @kubernetes(secrets="...") API.

    Fixes

    • When using --run-id-file, the file is now written prior to execution when resuming a flow (#1051). That matches how run command behaves already.

    Full Changelog: https://github.com/Netflix/metaflow/compare/2.6.1...2.6.2

    Source code(tar.gz)
    Source code(zip)
  • 2.6.1(May 13, 2022)

    Metaflow 2.6.1 Release Notes

    The Metaflow 2.6.1 release is a minor release.

    Features

    • Proper support for custom S3 endpoints. This enables using S3-compatible object storages like MinIO or Dell EMC-ECS as data stores for Metaflow ( https://github.com/Netflix/metaflow/pull/1045 )

    Bug fixes

    • Fixed card rendering for tables with some NaN values (https://github.com/Netflix/metaflow/issues/1023) in https://github.com/Netflix/metaflow/pull/1025
    • current.pathspec to return None when used outside Flow in https://github.com/Netflix/metaflow/pull/1033
    • Fixed bug in the card list command in https://github.com/Netflix/metaflow/pull/1044
    • Fixed issues with S3 get and ranges in https://github.com/Netflix/metaflow/pull/1034
    • Fix _new_task calling bug in LocalMetadataProvider in https://github.com/Netflix/metaflow/pull/1046

    Full Changelog: https://github.com/Netflix/metaflow/compare/2.6.0...2.6.1

    Source code(tar.gz)
    Source code(zip)
  • 2.6.0(Apr 25, 2022)

    Metaflow 2.6.0 Release Notes

    The Metaflow 2.6.0 release is a minor release and introduces Metaflow's integration with Kubernetes and Argo Workflows

    • Features
      • Add capability to launch Metaflow tasks on Kubernetes and schedule Metaflow flows with Argo Workflows.
      • Expose tags in current object.

    Features

    Add capability to launch Metaflow tasks on Kubernetes and schedule Metaflow flows with Argo Workflows.

    This release enables brand new capabilities for Metaflow on top of Kubernetes. You can now run --with kubernetes all or parts of any Metaflow flow on top of any Kubernetes cluster from your workstation. To execute your flow asynchronously, you can deploy the flow to Argo Workflows (a Kubernetes-native workflow scheduler) with a single command - argo-workflows create.

    To get started, take a look at the deployment guide for Kubernetes. Your feedback and feature requests are highly appreciated! - please reach out to us at slack.outerbounds.co

    PR #992 addressed issue #50.

    Expose tags in current object.

    Metaflow tags are now available as part of the current singleton object.

    @step
    def my_step(self):
        from metaflow import current
        tags = current.tags
        ...
    

    PR #1019 fixed issue #1007.

    Source code(tar.gz)
    Source code(zip)
  • 2.5.4(Mar 25, 2022)

    Metaflow 2.5.4 Release Notes

    The Metaflow 2.5.4 release is a minor release.

    Bug Fixes

    • Card bug fixes (https://github.com/Netflix/metaflow/pull/989, https://github.com/Netflix/metaflow/pull/981 )
    • importlib_metadata fixes for Python 3.5 (https://github.com/Netflix/metaflow/pull/988 )
    • Configurable temp root when pulling artifacts from s3 by @kgullikson88 in https://github.com/Netflix/metaflow/pull/991

    Full Changelog: https://github.com/Netflix/metaflow/compare/2.5.3...2.5.4

    Source code(tar.gz)
    Source code(zip)
  • 2.5.3(Mar 8, 2022)

    Metaflow 2.5.3 Release Notes

    The Metaflow 2.5.3 release is a minor release.

    Fixes

    • Fix "Too many symbolic links" error when using Conda + Batch on MacOS by @bishax in https://github.com/Netflix/metaflow/pull/972
    • Emit app tag for AWS Batch jobs ( https://github.com/Netflix/metaflow/pull/970 )

    Full Changelog: https://github.com/Netflix/metaflow/compare/2.5.2...2.5.3

    Source code(tar.gz)
    Source code(zip)
  • 2.5.2(Feb 17, 2022)

    Metaflow 2.5.2 Release Notes

    The Metaflow 2.5.2 release is a minor release.

    Improvements

    • follow symlinks when creating code packages https://github.com/Netflix/metaflow/pull/958

    Full Changelog: https://github.com/Netflix/metaflow/compare/2.5.1...2.5.2

    Source code(tar.gz)
    Source code(zip)
  • 2.5.1(Feb 15, 2022)

    Metaflow 2.5.1 Release Notes

    The Metaflow 2.5.1 release is a minor release.

    New Features

    • Introduce Mamba as a dependency solver for @conda in https://github.com/Netflix/metaflow/pull/918 . Mamba promises faster package dependency resolution times, which should result in an appreciable speedup in flow environment initialization. It is not yet enabled by default; to use it you need to set METAFLOW_CONDA_DEPENDENCY_RESOLVER to mamba in Metaflow config.

    Improvements

    • Vendor in click to reduce chances of dependency conflicts with user code in https://github.com/Netflix/metaflow/pull/929

    Full Changelog: https://github.com/Netflix/metaflow/compare/2.5.0...2.5.1

    Source code(tar.gz)
    Source code(zip)
  • 2.5.0(Jan 25, 2022)

    Metaflow 2.5.0 Release Notes

    The Metaflow 2.5.0 release is a minor release.

    New Features

    :sparkles: Metaflow cards are now publicly available! For details, see a new section in the documentation, Visualizing Results, and a release blog post.

    Bug Fixes

    • Fix issue in Step Functions integration with CLI defined decorators ( https://github.com/Netflix/metaflow/pull/920 )
    • Fix compute_resources to take into account string values ( https://github.com/Netflix/metaflow/pull/919 )

    Full Changelog: https://github.com/Netflix/metaflow/compare/2.4.9...2.5.0

    Source code(tar.gz)
    Source code(zip)
  • 2.4.9(Jan 19, 2022)

    Metaflow 2.4.9 Release Notes

    The Metaflow 2.4.9 release is a patch release.

    Improvements

    • Store information about the DAG being executed in an artifact. This will allow to render execution DAG in a @card ( https://github.com/Netflix/metaflow/pull/822 )

    Bug Fixes

    • Fixed cli command when task_id provided ( by @zhugejun in https://github.com/Netflix/metaflow/pull/890 )
    • Fix with metadata syncing on AWS Batch when running without remote metadata service ( https://github.com/Netflix/metaflow/pull/902 )
    • Fix default resource math. Previously we sometimes computed vCPU and memory settings incorrectly, in cases when they were set to something less than the default value ( https://github.com/Netflix/metaflow/pull/810 , fixes https://github.com/Netflix/metaflow/issues/467 )

    Full Changelog: https://github.com/Netflix/metaflow/compare/2.4.8...2.4.9

    Source code(tar.gz)
    Source code(zip)
  • 2.4.8(Jan 11, 2022)

    Metaflow 2.4.8 Release Notes

    The Metaflow 2.4.8 release is a patch release.

    Bug fixes

    • aws_retry's S3_RETRY_COUNT now has to be >=1 ( https://github.com/Netflix/metaflow/pull/876 )
    • fix argument type handling for host_volumes when used with --with and Step Functions ( https://github.com/Netflix/metaflow/pull/884 )

    Improvements

    • Improved validation logic to capture reserved keywords ( https://github.com/Netflix/metaflow/pull/830 fixes #589 )
    • Remove default use of repo.anaconda.com (https://github.com/Netflix/metaflow/pull/832 )

    Full Changelog: https://github.com/Netflix/metaflow/compare/2.4.7...2.4.8

    Source code(tar.gz)
    Source code(zip)
Owner
Netflix, Inc.
Netflix Open Source Platform
Netflix, Inc.
Kedro is an open-source Python framework for creating reproducible, maintainable and modular data science code

A Python framework for creating reproducible, maintainable and modular data science code.

QuantumBlack Labs 7.9k Jan 1, 2023
A logical, reasonably standardized, but flexible project structure for doing and sharing data science work.

Cookiecutter Data Science A logical, reasonably standardized, but flexible project structure for doing and sharing data science work. Project homepage

Jon C Cline 0 Sep 5, 2021
Data intensive science for everyone.

The latest information about Galaxy can be found on the Galaxy Community Hub. Community support is available at Galaxy Help. Galaxy Quickstart Galaxy

Galaxy Project 1k Jan 8, 2023
CS 506 - Computational Tools for Data Science

CS 506 - Computational Tools for Data Science Code, slides, and notes for Boston University CS506 Fall 2021 The Final Project Repository can be found

Lance Galletti 14 Mar 23, 2022
A framework for feature exploration in Data Science

Beehive A framework for feature exploration in Data Science Background What do we do when we finish one episode of feature exploration in a jupyter no

Steven IJ 1 Jan 3, 2022
ReproZip is a tool that simplifies the process of creating reproducible experiments from command-line executions, a frequently-used common denominator in computational science.

ReproZip ReproZip is a tool aimed at simplifying the process of creating reproducible experiments from command-line executions, a frequently-used comm

null 267 Jan 1, 2023
collection of interesting Computer Science resources

collection of interesting Computer Science resources

Kirill Bobyrev 137 Dec 22, 2022
PsychoPy is an open-source package for creating experiments in behavioral science.

PsychoPy is an open-source package for creating experiments in behavioral science. It aims to provide a single package that is: precise enoug

PsychoPy 1.3k Dec 31, 2022
Algorithms covered in the Bioinformatics Course part of the Cambridge Computer Science Tripos

Bioinformatics This is a repository of all the algorithms covered in the Bioinformatics Course part of the Cambridge Computer Science Tripos Algorithm

null 16 Jun 30, 2022
Datamol is a python library to work with molecules

Datamol is a python library to work with molecules. It's a layer built on top of RDKit and aims to be as light as possible.

datamol 276 Dec 19, 2022
PennyLane is a cross-platform Python library for differentiable programming of quantum computers.

PennyLane is a cross-platform Python library for differentiable programming of quantum computers. Train a quantum computer the same way as a neural network.

PennyLaneAI 1.6k Jan 4, 2023
3D visualization of scientific data in Python

Mayavi: 3D visualization of scientific data in Python Mayavi docs: http://docs.enthought.com/mayavi/mayavi/ TVTK docs: http://docs.enthought.com/mayav

Enthought, Inc. 1.1k Jan 6, 2023
Zipline, a Pythonic Algorithmic Trading Library

Zipline is a Pythonic algorithmic trading library. It is an event-driven system for backtesting. Zipline is currently used in production as the backte

Quantopian, Inc. 15.7k Jan 7, 2023
An interactive explorer for single-cell transcriptomics data

an interactive explorer for single-cell transcriptomics data cellxgene (pronounced "cell-by-gene") is an interactive data explorer for single-cell tra

Chan Zuckerberg Initiative 424 Dec 15, 2022
🍊 :bar_chart: :bulb: Orange: Interactive data analysis

Orange Data Mining Orange is a data mining and visualization toolbox for novice and expert alike. To explore data with Orange, one requires no program

Bioinformatics Laboratory 3.9k Jan 5, 2023
Incubator for useful bioinformatics code, primarily in Python and R

Collection of useful code related to biological analysis. Much of this is discussed with examples at Blue collar bioinformatics. All code, images and

Brad Chapman 560 Dec 24, 2022
Probabilistic Programming in Python: Bayesian Modeling and Probabilistic Machine Learning with Aesara

PyMC3 is a Python package for Bayesian statistical modeling and Probabilistic Machine Learning focusing on advanced Markov chain Monte Carlo (MCMC) an

PyMC 7.2k Dec 30, 2022
Statsmodels: statistical modeling and econometrics in Python

About statsmodels statsmodels is a Python package that provides a complement to scipy for statistical computations including descriptive statistics an

statsmodels 8.1k Dec 30, 2022
CONCEPT (COsmological N-body CodE in PyThon) is a free and open-source simulation code for cosmological structure formation

CONCEPT (COsmological N-body CodE in PyThon) is a free and open-source simulation code for cosmological structure formation. The code should run on any Linux system, from massively parallel computer clusters to laptops.

Jeppe Dakin 62 Dec 8, 2022