An end-to-end implementation of intent prediction with Metaflow and other cool tools

Overview

You Don't Need a Bigger Boat

An end-to-end (Metaflow-based) implementation of an intent prediction flow for kids who can't MLOps good and wanna learn to do other stuff good too.

This is a WIP - check back often for updates.

Philosophical Motivations

There is plenty of tutorials and blog posts around the Internet on data pipelines and tooling. However:

  • they (for good pedagogical reasons) tend to focus on one tool / step at a time, leaving us to wonder how the rest of the pipeline works;
  • they (for good pedagogical reasons) tend to work in a toy-world fashion, leaving us to wonder what would happen when a real dataset and a real-world problem enter the scene.

This repository (and soon-to-be-drafted written tutorial) aims to fill these gaps. In particular:

  • we provide open-source working code that glues together what we believe are some of the best tools in the ecosystem, going all the way from raw data to a deployed endpoint serving predictions;
  • we run the pipeline under a realistic load for companies at "reasonable scale", leveraging a huge open dataset we released in 2021; moreover, we train a model for a real-world use case, and show how to monitor it after deployment.

The repo may also be seen as a (very opinionated) introduction to modern, PaaS-like pipelines; while there is obviously room for disagreement over tool X or tool Y, we believe the general principles to be sound for companies at "reasonable scale": in-between bare-bone infrastructure for Tech Giants, and ready-made solutions for low-code/simple scenarios, there is a world of exciting machine learning at scale for sophisticated practitioners who don't want to waste their time managing cloud resources.

Overview

The repo shows how several (mostly open-source) tools can be effectively combined together to run data pipelines. The project current features:

The following picture from our Recsys paper (forthcoming) gives a quick overview of such a pipeline:

Recsys flow

We provide two versions of the pipeline, depending on the sophistication of the setup:

  • a Metaflow-only version, which runs from static data files (see below) to Sagemaker as a single Flow, and can be run from a Metaflow-enabled laptop without much additional setup;
  • a data warehouse version, which runs in a more realistic setup, reading data from Snowflake and using an external orchestrator to run the steps. In this setup, the downside is that a Snowflake and a Prefect Cloud accounts are required (nonetheless, both are veasy to get); the upside is that the pipeline reflects almost perfectly a real setup, and Metaflow can be used specifically for the ML part of the process.

The parallelism between the two scenarios should be pretty clear by looking at the two projects: if you are familiarizing with all the tools for the first time, we suggest you to start from the Metaflow version and then move to the full-scale one when all the pieces of the puzzle are well understood.

Relevant Material

If you want to know more, you can give a look at the following material:

TBC

Status Update

July 2021

End-2-end flow working for remote and local projects; started standardizing Prefect agents with Docker and adding other services (monitoring, feature store etc.).

TO-DOs:

  • dockerize the local flow;
  • write-up all of this as a blog post;
  • improve code / readability / docs, add potentially some more pics and some videos;
  • providing an orchestrator-free version, by using step functions to manage the steps;
  • finish feature store and gantry integration;
  • add Github Action flow;
  • continue improving the DAG card project.

Setup

General Prerequisites (do this first!)

Irrespectively of the flow you wish to run, some general tools need to be in place: Metaflow of course, as the heart of our ML practice, but also data and AWS users/roles. Please go through the general items below before tackling the flow-specific instructions.

After you finish the prerequisites below, you can run the flow you desire: each folder - remote and local - contains a specific README which should allow you to quickly run the project end-to-end: please refer to that documentation for flow-specific instructions (check back often for updates).

Dataset

The project leverages the open dataset from the 2021 Coveo Data Challenge: the dataset can be downloaded directly from here (refer to the full README for terms and conditions). Data is freely available under a research-friendly license - for background information on the dataset, the use cases and relevant work in the ML literature, please refer to the accompanying paper.

Once you download and unzip the dataset in a local folder of your choice (the zip contains 3 files, browsing_train.csv, search_train.csv, sku_to_content.csv), write down their location as an absolute path (e.g. /Users/jacopo/Documents/data/train/browsing_train.csv): both projects need to know where the dataset is.

AWS

Both projects - remote and local - use AWS services extensively - and by design: this ties back to our philosophy of PaaS-whenever-possible, and play nicely with our core adoption of Metaflow. While you can setup your users in many functionally equivalent ways, note that if you want to run the pipeline from ingestion to serving you need to be comfortable with the following AWS interactions:

  • Metaflow stack (see below): we assume you installed the Metaflow stack and can run it with an AWS profile of your choice;
  • Serverless stack (see below): we assume you can run serverless deploy in your AWS stack;
  • Sagemaker user: we assume you have an AWS user with permissions to manage Sagemaker endpoints (it may be totally distinct from any other Metaflow user).

TBC

Serverless

We wrap Sagemaker predictions in a serverless REST endpoint provided by AWS Lambda and API Gateway. To manage the lambda stack we use Serverless as a wrapper around AWS infrastructure.

TBC

Metaflow

Metaflow: Configuration

If you have an AWS profile configured with a metaflow-friendly user, and you created metaflow stack with CloudFormation, you can run the following command with the resources created by CloudFormation to set up metaflow on AWS:

metaflow configure aws --profile metaflow

Remember to use METAFLOW_PROFILE=metaflow to use this profile when running a flow. Once you completed the setup, you can run flow_playground.py to test the AWS setup is working as expected (in particular, GPU batch jobs can run correctly). To run the flow with the custom profile created, you should do:

METAFLOW_PROFILE=metaflow python flow_playground.py run

Metaflow: Tips & Tricks
  1. Parallelism Safe Guard
    • The flag --max-workers should be used to limit the maximum number of parallel steps
    • For example METAFLOW_PROFILE=metaflow python flow_playground.py run --max-workers 8 limits the maximum number of parallel tasks to 8
  2. Environment Variables in AWS Batch
    • The @environment decorator is used in conjunction with @batch to pass environment variables to AWS Batch, which will not directly have access to env variables on your local machine
    • In the local example, we use @environemnt to pass the Weights & Biases API Key (amongst other things)
  3. Resuming Flows
    • Resuming flows is useful during development to avoid re-running compute/time intensive steps such as data preparation
    • METAFLOW_PROFILE=metaflow python flow_playground.py resume <STEP_NAME> --origin-run-id <RUN_ID>
  4. Local-Only execution
    • It may sometimes be useful to debug locally (i.e to avoid Batch startup latency), we introduce a wrapper enable_decorator around the @batch decorator which enables or disables a decorator's functionality
    • We use this in conjunction with an environment variable EN_BATCH to toggle the functionality of all @batch decorators.

FAQ

  1. Both projects deal with data that has already been ingested/transmitted to the pipeline, but are silent on data collection. Any serverless option there as well?

    Yes. In e-commerce use cases, for example, pixel tracking is standard (e.g. Google Analytics), so a serverless /collect endpoint can be used to get front-end data and drop it in a pure PaaS pipeline with Firehose and Snowpipe, for example. While a bit out-dated for some details, we championed exactly this approach a while ago: if you want to know more, you can start from this Medium post and old code.

TBC

How to Cite our Work

If you find our principles, code or data useful, please cite our work:

Paper (forthcoming in RecSys2021)

@inproceedings{10.1145/3460231.3474604,
author = {Tagliabue, Jacopo},
title = {You Do Not Need a Bigger Boat: Recommendations at Reasonable Scale in a (Mostly) Serverless and Open Stack},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3460231.3474604},
doi = {10.1145/3460231.3474604},
series = {RecSys '21}
}

Data

@inproceedings{CoveoSIGIR2021,
author = {Tagliabue, Jacopo and Greco, Ciro and Roy, Jean-Francis and Bianchi, Federico and Cassani, Giovanni and Yu, Bingqing and Chia, Patrick John},
title = {SIGIR 2021 E-Commerce Workshop Data Challenge},
year = {2021},
booktitle = {SIGIR eCom 2021}
}
Comments
  • Open source options for all stages

    Open source options for all stages

    Thanks a lot for your work. Love your repos and podcasts. I wanted to suggest adding atleast one open source alternative for each stage. It would be helpful for folks that was to quickly build a full system with freely available open source tools. Thanks!

    opened by bsridatta 2
  • Running local flows without GPUs

    Running local flows without GPUs

    hi all!

    Thank you for these incredible resources :)

    how would I run the local flow if I don't have access to any GPUs?

    https://github.com/jacopotagliabue/you-dont-need-a-bigger-boat/tree/main/local_flow

    opened by hugobowne 2
  • Fix missing ESCAPED_DQ File Format

    Fix missing ESCAPED_DQ File Format

    The definition of the custom ESCAPED_DQ file format is missing on the sf_connector file, causing issues when a user is trying to upload the data into Snowflake.

    Original issue: https://github.com/jacopotagliabue/you-dont-need-a-bigger-boat/issues/6

    opened by bigluck 2
  • File Format Does Not Exist

    File Format Does Not Exist

    Hey there!

    I was recently looking at the remote_flow workflow in the repo. However at the step where data needs to be uploaded into a Snowflake instance, I have been running into an error when calling make upload:

    Traceback (most recent call last):
      File "push_data_to_sf.py", line 83, in <module>
        write_chunks(table=sku_to_content_table,
      File "push_data_to_sf.py", line 64, in write_chunks
        conn.upload_file(f"{output_prefix}*", table_name)
      File "/app/connectors/sf_connector.py", line 79, in upload_file
        self._cs.execute(f"COPY INTO {table} FILE_FORMAT = ESCAPED_DQ")
      File "/usr/local/lib/python3.8/site-packages/snowflake/connector/cursor.py", line 693, in execute
        Error.errorhandler_wrapper(
      File "/usr/local/lib/python3.8/site-packages/snowflake/connector/errors.py", line 258, in errorhandler_wrapper
        cursor.errorhandler(connection, cursor, error_class, error_value)
      File "/usr/local/lib/python3.8/site-packages/snowflake/connector/errors.py", line 188, in default_errorhandler
        raise error_class(
    snowflake.connector.errors.ProgrammingError: 002003 (02000): SQL compilation error:
    File format 'ESCAPED_DQ' does not exist or not authorized.
    

    It appears that ESCAPED_DQ might be a custom defined file format for Snowflake data ingestion? Where is its definition?

    Thanks for putting this repo together!

    opened by mihail911 2
  • only enable pip install decorator if batch enabled

    only enable pip install decorator if batch enabled

    I wasn't running your exact example but I was having issues with running this locally with the pip install decorator. I think it should also be wrapped and only run on batch. ❤️ the clever decorators to install on batch.

    opened by JSpenced 2
  • Comparison between this and something like Kedro or ZenML?

    Comparison between this and something like Kedro or ZenML?

    Hey there! Thanks for this amazing work! I was wondering if you /any users here had done a comparison between these projects. Seems like this repo just describes a general set of tools and how they link up together whereas the others take an opinionated stance and do the linking themselves?

    I'm in the process of evaluating tools / frameworks.

    I'm currently a single person looking to set up the groundwork for things to come. So far my plans have been to stay local for as long as possible before moving to some distributed computing framework (Dask gave me a lot of trouble in the past). I'm also looking to avoid using tools such as AWS or GCP for as long as possible so ideally the discussion would revolve around local machines. I'd love to hear thoughts and opinions.

    opened by IanQS 1
  • only pip install packages if running on batch

    only pip install packages if running on batch

    I accidentally closed the other pull requests.

    This will only pip install packages if on batch. I wasn't sure if to put this at this level or move it before the wrapper decorator.

    opened by JSpenced 0
  • Andrew/standarization of variables

    Andrew/standarization of variables

    Changes:

    • Updates to the variable names for SQL

    and

    • using serverless-dotenv-plugin

    The plugin allows us to populate the serverless.yml with the contents of the .env file. It seemed to be the approach with less friction.

    opened by asutcliffe-coveo 0
  • Andrew/dbt

    Andrew/dbt

    This is a basic setup for DBT, the project structure could be review. Currently sigir_dbt is under prototype_flow a the same level as source which contains my python scripts to unprocess data.

    The README in sigir_dbt should contain the information needed for setup. Ping me if you have an issue. If everything works on your side we could merge.

    Important notes:

    • If you want to se the python connector for snowflake we may need to fix the version for pyarrow as it could lead to conflicts with dbt. I remeber gettign a warning.
    • Also I will do a clean up of the src folder and merge it in tonight.
    opened by asutcliffe-coveo 0
Owner
Jacopo Tagliabue
I failed the Turing Test once, but that was many friends ago.
Jacopo Tagliabue
Pytorch implementation of "Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling"

RNN-for-Joint-NLU Pytorch implementation of "Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling"

Kim SungDong 194 Dec 28, 2022
GPU-accelerated PyTorch implementation of Zero-shot User Intent Detection via Capsule Neural Networks

GPU-accelerated PyTorch implementation of Zero-shot User Intent Detection via Capsule Neural Networks This repository implements a capsule model Inten

Joel Huang 15 Dec 24, 2022
(CVPR 2022) A minimalistic mapless end-to-end stack for joint perception, prediction, planning and control for self driving.

LAV Learning from All Vehicles Dian Chen, Philipp Krähenbühl CVPR 2022 (also arXiV 2203.11934) This repo contains code for paper Learning from all veh

Dian Chen 300 Dec 15, 2022
Open source code for Paper "A Co-Interactive Transformer for Joint Slot Filling and Intent Detection"

A Co-Interactive Transformer for Joint Slot Filling and Intent Detection This repository contains the PyTorch implementation of the paper: A Co-Intera

null 67 Dec 5, 2022
SlotRefine: A Fast Non-Autoregressive Model forJoint Intent Detection and Slot Filling

SlotRefine: A Fast Non-Autoregressive Model for Joint Intent Detection and Slot Filling Reference Main paper to be cited (Di Wu et al., 2020) @article

Moore 34 Nov 3, 2022
Intent parsing and slot filling in PyTorch with seq2seq + attention

PyTorch Seq2Seq Intent Parsing Reframing intent parsing as a human - machine translation task. Work in progress successor to torch-seq2seq-intent-pars

Sean Robertson 160 Jan 7, 2023
pytorch bert intent classification and slot filling

pytorch_bert_intent_classification_and_slot_filling 基于pytorch的中文意图识别和槽位填充 说明 基本思路就是:分类+序列标注(命名实体识别)同时训练。 使用的预训练模型:hugging face上的chinese-bert-wwm-ext 依

西西嘛呦 33 Dec 15, 2022
On-device speech-to-intent engine powered by deep learning

Rhino Made in Vancouver, Canada by Picovoice Rhino is Picovoice's Speech-to-Intent engine. It directly infers intent from spoken commands within a giv

Picovoice 510 Dec 30, 2022
Citation Intent Classification in scientific papers using the Scicite dataset an Pytorch

Citation Intent Classification Table of Contents About the Project Built With Installation Usage Acknowledgments About The Project Citation Intent Cla

Federico Nocentini 4 Mar 4, 2022
Checkout some cool self-projects you can try your hands on to curb your boredom this December!

SoC-Winter Checkout some cool self-projects you can try your hands on to curb your boredom this December! These are short projects that you can do you

Web and Coding Club, IIT Bombay 29 Nov 8, 2022
Have you ever wondered how cool it would be to have your own A.I

Have you ever wondered how cool it would be to have your own A.I. assistant Imagine how easier it would be to send emails without typing a single word, doing Wikipedia searches without opening web browsers, and performing many other daily tasks like playing music with the help of a single voice command.

Harsh Gupta 1 Nov 9, 2021
A cool little repl-based simulation written in Python

A cool little repl-based simulation written in Python planned to integrate machine-learning into itself to have AI battle to the death before your eye

Em 6 Sep 17, 2022
A pure PyTorch batched computation implementation of "CIF: Continuous Integrate-and-Fire for End-to-End Speech Recognition"

A pure PyTorch batched computation implementation of "CIF: Continuous Integrate-and-Fire for End-to-End Speech Recognition"

張致強 14 Dec 2, 2022
Price-Prediction-For-a-Dream-Home - A machine learning based linear regression trained model for house price prediction.

Price-Prediction-For-a-Dream-Home ROADMAP TO THIS LINEAR REGRESSION BASED HOUSE PRICE PREDICTION PREDICTION MODEL Import all the dependencies of the p

DIKSHA DESWAL 1 Dec 29, 2021
Doge-Prediction - Coding Club prediction ig

Doge-Prediction Coding Club prediction ig Basically: Create an application that

null 1 Jan 10, 2022
PyTorch implementation of the end-to-end coreference resolution model with different higher-order inference methods.

End-to-End Coreference Resolution with Different Higher-Order Inference Methods This repository contains the implementation of the paper: Revealing th

Liyan 52 Jan 4, 2023
An implementation for `Text2Event: Controllable Sequence-to-Structure Generation for End-to-end Event Extraction`

Text2Event An implementation for Text2Event: Controllable Sequence-to-Structure Generation for End-to-end Event Extraction Please contact Yaojie Lu (@

Roger 153 Jan 7, 2023
This repository is an official implementation of the paper MOTR: End-to-End Multiple-Object Tracking with TRansformer.

MOTR: End-to-End Multiple-Object Tracking with TRansformer This repository is an official implementation of the paper MOTR: End-to-End Multiple-Object

null 348 Jan 7, 2023
PyTorch implementation of SampleRNN: An Unconditional End-to-End Neural Audio Generation Model

samplernn-pytorch A PyTorch implementation of SampleRNN: An Unconditional End-to-End Neural Audio Generation Model. It's based on the reference implem

DeepSound 261 Dec 14, 2022