Event-driven-model-serving - Unified API of Apache Kafka and Google PubSub

Overview

event-driven-model-serving

Unified API of Apache Kafka and Google PubSub

1. Project Structure

.event-driven-model-serving
+-- Modelling
|   +-- model_checkpoints
|       +-- mnist_cnn_model.h5
|   +-- plots
|       +-- loss_plot.png
|
|   +-- __init__.py
|   +-- data_preparation.py
|   +-- train_cnn.py
|   +-- predict_cnn.py
|
+-- MessageBroker
|   +-- __init__.py
|   +-- broker_interface.py
|   +-- google_pubsub_handler.py
|   +-- kafka_handler.py
|   +-- message_broker_factory.py
|   +-- kafka_config.ini
|   +-- pubsub_config.ini
|
+-- Queue
|   +-- log.txt
|
+-- DB
|   +-- db_for_app_2.txt
|
+-- app_1.py
+-- model_server_sub.py
+-- model_server_pub.py
+-- app_2.py

2. Cloud service configuration

2-1. Confluent Kafka

  1. In MessageBroker/kafka_config.ini, fill in bootstrap.servers, sasl.username, and sasl.password.

  2. Create two topics in your Kafka cluster. Name one as new_image_topic, another as inference_topic.

    Note:
    There is a create_topic() function in each handler, which are not working yet. I will make an update on this soon. For now, you can creat topics by manually in the confluent console.

2-2. Google Pub/Sub

  1. In MessageBroker/pubsub_config.ini, fill in your project_id.

    Note:
    If you are using gcloud CLI, after you set you project with gcloud config set project <Your project name>, project_id can be found by gcloud config get-value project.

  2. Set up Service account and Cloud IAM role.

    Go to https://console.cloud.google.com/iam-admin/ and pick "service account" tab.
    Make one and create JSON key.

    Then set a environment variable with your JSON key like below:
    export GOOGLE_APPLICATION_CREDENTIALS=~/Downloads/pubsub-trial-key.json

  3. Like you did in Kafka, please make two topics.
    Name one as new_image_topic, another as inference_topic.

    In GCP console, when you make these topic, check Add a default subscription as an option. This will create two according subscription_id named, new_image_topic-sub and inference_topic-sub

Every thing is ready for the cloud configuration!

3. About the architecture

  1. Queue directory.

    In the log.txt file, I simulate Redis Queue and FIFO strategy.
    This is to get results queued here before sending the results to application 2.

  2. DB directory.

    In the db_for_app_2.txt file, I simulate a simple database for application 2.
    This shows that application 2 successfully pull result from model server's publisher.

  3. BrokerInterface class is a abstract base class that makes interface for two handlers: KafkaHandler and PubSubHandler.

  4. MessageBrokerFactory creates instance object of either two Handler class.
    Default is set to KafkaHandler.

  5. However, no matter what Handler is created, this unified API ensures the same method names, parameters, and the behaviours.

Side note

  • Currently, the image recognition model is only trained with 28x28 pixel grey scale image. It does not support different size nor color image.
You might also like...
An API-driven solution for Makerspaces, Tinkerers, and Hackers.
An API-driven solution for Makerspaces, Tinkerers, and Hackers.

Mventory is an API-driven inventory solution for Makers, Makerspaces, Hackspaces, and just about anyone else who needs to keep track of "stuff".

Mventory is an API-driven solution for Makerspaces, Tinkerers, and Hackers.
Mventory is an API-driven solution for Makerspaces, Tinkerers, and Hackers.

Mventory is an API-driven inventory solution for Makers, Makerspaces, Hackspaces, and just about anyone else who needs to keep track of "stuff".

An attendance bot that joins google meet automatically according to schedule and marks present in the google meet.
An attendance bot that joins google meet automatically according to schedule and marks present in the google meet.

Google-meet-self-attendance-bot An attendance bot which joins google meet automatically according to schedule and marks present in the google meet. I

Google Drive, OneDrive and Youtube as covert-channels - Control systems remotely by uploading files to Google Drive, OneDrive, Youtube or Telegram
Google Drive, OneDrive and Youtube as covert-channels - Control systems remotely by uploading files to Google Drive, OneDrive, Youtube or Telegram

covert-control Control systems remotely by uploading files to Google Drive, OneDrive, Youtube or Telegram using Python to create the files and the lis

Automation that uses Github Actions, Google Drive API, YouTube Data API and youtube-dl together to feed BackJam app with new music

Automation that uses Github Actions, Google Drive API, YouTube Data API and youtube-dl together to feed BackJam app with new music

The Official Twilio SendGrid Led, Community Driven Python API Library
The Official Twilio SendGrid Led, Community Driven Python API Library

The default branch name for this repository has been changed to main as of 07/27/2020. This library allows you to quickly and easily use the SendGrid

Google scholar share - Simple python script to pull Google Scholar data from an author's profile

google_scholar_share Simple python script to pull Google Scholar data from an au

AWS Glue PySpark - Apache Hudi Quick Start Guide

AWS Glue PySpark - Apache Hudi Quick Start Guide Disclaimer: This is a quick start guide for the Apache Hudi Python Spark connector, running on AWS Gl

Owner
Danny Toeun Kim
MEng. Computer Science at @UCL
Danny Toeun Kim
42-event-notifier - 42 Event notifier using 42API and Github Actions

42 Event Notifier 42서울 Agenda에 새로운 이벤트가 등록되면 알려드립니다! 현재는 Github Issue로 등록되므로 상단

null 6 May 16, 2022
First Party data integration solution built for marketing teams to enable audience and conversion onboarding into Google Marketing products (Google Ads, Campaign Manager, Google Analytics).

Megalista Sample integration code for onboarding offline/CRM data from BigQuery as custom audiences or offline conversions in Google Ads, Google Analy

Google 76 Dec 29, 2022
An API serving data on all creatures, monsters, materials, equipment, and treasure in The Legend of Zelda: Breath of the Wild

Hyrule Compendium API An API serving data on all creatures, monsters, materials, equipment, and treasure in The Legend of Zelda: Breath of the Wild. B

Aarav Borthakur 116 Dec 1, 2022
Fetching tweets and integrating it with Kafka and PySpark

KafkaPySpark Zookeeper bin/zookeeper-server-start.sh config/zookeeper.properties Kafka Server bin/kafka-server-start.sh config/server.properties Kafka

Priyansh 2 Dec 29, 2021
A simple healthcheck wrapper to monitor Kafka.

kafka-healthcheck A simple healthcheck wrapper to monitor Kafka. Kafka Healthcheck is a simple server that provides a singular API endpoint to determi

Rodrigo Nicolas Garcia 3 Oct 17, 2022
A simple bot that lives in your Telegram group, logging messages to a Postgresql database and serving statistical tables and plots to users as Telegram messages.

telegram-stats-bot Telegram-stats-bot is a simple bot that lives in your Telegram group, logging messages to a Postgresql database and serving statist

null 22 Dec 26, 2022
Easy Google Translate: Unofficial Google Translate API

easygoogletranslate Unofficial Google Translate API. This library does not need an api key or something else to use, it's free and simple. You can eit

Ahmet Eren Odacı 9 Nov 6, 2022
Many discord bots serving different purposes

Discord_Botlari Farklı amaçlara hizmet eden bir çok discord botu En kapsamlı Bot Game Bottur. bir oyun botudur discord sunucularında kullanılır. (tüm

null 1 Dec 21, 2021
A unified API wrapper for YouTube and Twitch chat bots.

Chatto A unified API wrapper for YouTube and Twitch chat bots. Contributing Chatto is open to contributions. To find out where to get started, have a

Ethan Henderson 5 Aug 1, 2022
DIAL(Did I Alert Lambda?) is a centralised security misconfiguration detection framework which completely runs on AWS Managed services like AWS API Gateway, AWS Event Bridge & AWS Lambda

DIAL(Did I Alert Lambda?) is a centralised security misconfiguration detection framework which completely runs on AWS Managed services like AWS API Gateway, AWS Event Bridge & AWS Lambda

CRED 71 Dec 29, 2022