Demonstration of the Model Training as a CI/CD System in Vertex AI

Last update: Dec 28, 2022

Overview

Model Training as a CI/CD System

This project demonstrates the machine model training as a CI/CD system in GCP platform. You will see more detailed workflow in the below section, but it is about rebuilding and redeploying (continuous integration) the currently deployed machine learning pipeline based on changes in code. Such changes could happen in the training data, data pre-processing logic, model architecture and training code, custom pipeline components, and so on.

Workflow #1

We create initial code, or we make some changes in the existing codebase for pipeline.
Based on the changes in the step 2, a GitHub action gets triggered to initiate a Cloud Build process.
The Cloud Build runs unit tests to see if those components work without errors.
If there is no error at all, there are two common sub-workflows from this point.
- Cloud Build containerizes the current codebase. This is an optional step. If you have any custom components unchanges, this step might be omitted.
  - The Cloud Build compiles a new pipeline. It creates an updated docker image, and it uploads the new docker image to GCR
- If there is any codes changed in data preprocessing, modeling, training steps, we only have to upload those source files to designated GCS bucket
The final step of the Cloud Build is to execute a pipeline run on Vertex AI

Workflow #2

Workflow in a nutshell

We create initial code, or we make some changes in the existing codebase for modules.
Based on the changes in the step 2, a GitHub action gets triggered to initiate a Cloud Build process.
The Cloud Build runs unit tests to see if those components work without errors.
If there is no error at all, there are two common sub-workflows from this point.
- If there is any codes changed in data preprocessing and models, we only have to upload those source files to designated GCS bucket.
The final step of the Cloud Build is to execute a pipeline run on Vertex AI. Trainer and Transform TFX components will look up the changed modules accordingly.

Acknowledgements

ML-GDE program for providing GCP credits.

Comments

Creating pipeline

Hi, I created a project in vertex by using the Taxi template.

$ tfx pipeline create \
      --pipeline-path=kubeflow_v2_runner.py \
      --engine=vertex \
      --build-image

...then i had this error:

requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/distribution/gcr.io/teco-prod-adam-dev-826c/tfx-pipeline/json

Any clue about that?...thanks in advance

opened by hugoferrero 8

Notebook for collating different steps for post #1
@deep-diver I can work on a notebook that could basically enumerate all the steps we are following to cover the first part:

Create a simple image with KFP that is used to create the template.

Mention that for the first time when the code is committed GitHub Actions will run the entire thing.

Clone this repository and commit a minimal change to the example/my-pipeline/ to trigger a run.

Let me know your thoughts and we can proceed accordingly.
opened by sayakpaul 3

BERT model training impelmentation using 1024 A100 GPUs for MLPerf Training v1.1

Pre-trained checkpoint and bert config json file Location of checkpoint and bert config json file This MLCommons members Google Drive location contain

SAIT (Samsung Advanced Institute of Technology)

12 Apr 27, 2022

In this project we investigate the performance of the SetCon model on realistic video footage. Therefore, we implemented the model in PyTorch and tested the model on two example videos.

Contrastive Learning of Object Representations Supervisor: Prof. Dr. Gemma Roig Institutions: Goethe University CVAI - Computational Vision & Artifici

6 Dec 8, 2022

Step by Step on how to create an vision recognition model using LOBE.ai, export the model and run the model in an Azure Function

3 Mar 30, 2022

Product-based-recommendation-system - A product based recommendation system which uses Machine learning algorithm such as KNN and cosine similarity

Product-based-recommendation-system A product based recommendation system which

2 Feb 15, 2022

Simple-System-Convert--C--F - Simple System Convert With Python

Simple-System-Convert--C--F REQUIREMENTS Python version : 3 HOW TO USE Run the c

2 Feb 16, 2022

Real-Time-Student-Attendence-System - Real Time Student Attendence System

Real-Time-Student-Attendence-System The Student Attendance Management System Pro

1 Feb 15, 2022

Demonstration of the Model Training as a CI/CD System in Vertex AI

Related tags

Overview

Model Training as a CI/CD System

Workflow #1

Workflow #2

Workflow in a nutshell

Acknowledgements

You might also like...

BERT model training impelmentation using 1024 A100 GPUs for MLPerf Training v1.1

In this project we investigate the performance of the SetCon model on realistic video footage. Therefore, we implemented the model in PyTorch and tested the model on two example videos.

Step by Step on how to create an vision recognition model using LOBE.ai, export the model and run the model in an Azure Function

Complete system for facial identity system

Face-Recognition-based-Attendance-System - An implementation of Attendance System in python.

Face-Recognition-Attendence-System - This face recognition Attendence system using Python

Product-based-recommendation-system - A product based recommendation system which uses Machine learning algorithm such as KNN and cosine similarity

Simple-System-Convert--C--F - Simple System Convert With Python

Real-Time-Student-Attendence-System - Real Time Student Attendence System

Comments

Creating pipeline

Notebook for collating different steps for post #1

Owner

Chansung Park

A small demonstration of using WebDataset with ImageNet and PyTorch Lightning

A small demonstration of using WebDataset with ImageNet and PyTorch Lightning

Public implementation of "Learning from Suboptimal Demonstration via Self-Supervised Reward Regression" from CoRL'21

robomimic: A Modular Framework for Robot Learning from Demonstration

Predicting path with preference based on user demonstration using Maximum Entropy Deep Inverse Reinforcement Learning in a continuous environment

Modified fork of Xuebin Qin's U-2-Net Repository. Used for demonstration purposes.

A TensorFlow implementation of Neural Program Synthesis from Diverse Demonstration Videos

E2e music remastering system - End-to-end Music Remastering System Using Self-supervised and Adversarial Training

Complete system for facial identity system. Include one-shot model, database operation, features visualization, monitoring

This is the code for our KILT leaderboard submission to the T-REx and zsRE tasks. It includes code for training a DPR model then continuing training with RAG.