โšพ๐Ÿค–โšพ Automatic baseball pitching overlay in realtime

Overview

โšพ Automatically overlaying pitch motion and trajectory with machine learning!

This project takes your baseball pitching clips and automatically generates the overlay. The input pitching clip could be directly from your phone or camera. The release point will be automatically detected by the program. This system will trace the trajectory and align all the videos to generate the overlay.

A fine-tuned YOLOv4 model is used to get the location of the ball. Then, I implemented SORT tracking algorithm to keep track of each individual ball. Lastly, I have applied some image registration techniques to deal with slight camera shift on each clip.

I'm still trying to improve it! Feel free to follow this project, also check out the Todo list.

The idea came from this incredible overlay.

๐Ÿ’ป Getting Started

These instructions will get you a copy of the project, and generates your own pitching overlay clip!

Get a copy

Get a copy of this project by simply running the git clone command.

git clone https://github.com/chonyy/ML-auto-baseball-pitching-overlay.git

Prerequisites

Before running the project, we have to install all the dependencies from requirements.txt

pip install -r requirements.txt

Overlay!

Last, run the project with your own clips!

Try a sample

python pitching_overlay.py

Try with yout own clips

Place your pitching videos in a folder, then specify the path in the CLI.

python pitching_overlay.py --videos_folder "./videos/videos"

๐Ÿ”จ Project Structure

๐ŸŽฌ More Demo

โ˜‘๏ธ Todo

  • Implement image registration to deal with camera shift
  • Build a demo web app for people to use it in realtime on web
  • Enable custom parameter tuning
  • Improve the visual effect
  • Write a Medium post to explain the technical workflow
  • Draw a structure diagram
You might also like...
A GPU-optional modular synthesizer in pytorch, 16200x faster than realtime, for audio ML researchers.
A GPU-optional modular synthesizer in pytorch, 16200x faster than realtime, for audio ML researchers.

torchsynth The fastest synth in the universe. Introduction torchsynth is based upon traditional modular synthesis written in pytorch. It is GPU-option

Realtime segmentation with ENet, the fast and accurate segmentation net.
Realtime segmentation with ENet, the fast and accurate segmentation net.

Enet This is a realtime segmentation net with almost 22 fps on GTX1080 ti, and the model size is very small with only 28M. This repo contains the infe

Yolov5+SlowFast:  Realtime Action Detection Based on PytorchVideo
Yolov5+SlowFast: Realtime Action Detection Based on PytorchVideo

Yolov5+SlowFast: Realtime Action Detection A realtime action detection frame work based on PytorchVideo. Here are some details about our modification:

Much faster than SORT(Simple Online and Realtime Tracking), a little worse than SORT

QSORT QSORT(Quick + Simple Online and Realtime Tracking) is a simple online and realtime tracking algorithm for 2D multiple object tracking in video s

Realtime YOLO Monster Detection With Non Maximum Supression
Realtime YOLO Monster Detection With Non Maximum Supression

Realtime-YOLO-Monster-Detection-With-Non-Maximum-Supression Table of Contents In

Facial Expression Detection In The Realtime
Facial Expression Detection In The Realtime

The human's facial expressions is very important to detect thier emotions and sentiment. It can be very efficient to use to make our computers make interviews. Furthermore, we have robots now can detect the human's emotions and based on thats take an action .etc. So, It will be better to provide a tool or model for this.

Realtime micro-expression recognition using OpenCV and PyTorch
Realtime micro-expression recognition using OpenCV and PyTorch

Micro-expression Recognition Realtime micro-expression recognition from scratch using OpenCV and PyTorch Try it out with a webcam or video using the e

Official implementation of the RAVE model: a Realtime Audio Variational autoEncoder
Official implementation of the RAVE model: a Realtime Audio Variational autoEncoder

RAVE: Realtime Audio Variational autoEncoder Official implementation of RAVE: A variational autoencoder for fast and high-quality neural audio synthes

Python library to receive live stream events like comments and gifts in realtime from TikTok LIVE.
Python library to receive live stream events like comments and gifts in realtime from TikTok LIVE.

TikTokLive A python library to connect to and read events from TikTok's LIVE service A python library to receive and decode livestream events such as

Comments
  • Error while installing requirements

    Error while installing requirements

    Hello, could you please land me a hand?

    I'am getting this kind of issue:

    ImportError: cannot import name 'soft_unicode' from 'markupsafe' (full traceback below)

    Googling this error I found that there's a mismatch in package dependencies, mainly the markupsafe library. I played around with various versions of the packages but didn't get it in an functional state.

    I tried to install it on Python 3.7, 3.8 and 3.10

    Can you please update the requirements file or give a hint how to install it?

    Thanks, Marek

    ` ERROR: Command errored out with exit status 1: command: /Users/jardik/Documents/overlay/ML-auto-baseball-pitching-overlay/env-MLauto/bin/python3 /Users/jardik/Documents/overlay/ML-auto-baseball-pitching-overlay/env-MLauto/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /var/folders/vx/5x_d29f50t7387_bmsjsc8700000gn/T/tmp8cv3k60q cwd: /private/var/folders/vx/5x_d29f50t7387_bmsjsc8700000gn/T/pip-install-g_jmq3w8/astropy Complete output (28 lines): Traceback (most recent call last): File "/Users/jardik/Documents/overlay/ML-auto-baseball-pitching-overlay/env-MLauto/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 207, in main() File "/Users/jardik/Documents/overlay/ML-auto-baseball-pitching-overlay/env-MLauto/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 197, in main json_out['return_val'] = hook(**hook_input['kwargs']) File "/Users/jardik/Documents/overlay/ML-auto-baseball-pitching-overlay/env-MLauto/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 54, in get_requires_for_build_wheel return hook(config_settings) File "/private/var/folders/vx/5x_d29f50t7387_bmsjsc8700000gn/T/pip-build-env-kxuy583w/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 177, in get_requires_for_build_wheel return self._get_build_requires( File "/private/var/folders/vx/5x_d29f50t7387_bmsjsc8700000gn/T/pip-build-env-kxuy583w/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 159, in _get_build_requires self.run_setup() File "/private/var/folders/vx/5x_d29f50t7387_bmsjsc8700000gn/T/pip-build-env-kxuy583w/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 174, in run_setup exec(compile(code, file, 'exec'), locals()) File "setup.py", line 70, in ext_modules=get_extensions()) File "/private/var/folders/vx/5x_d29f50t7387_bmsjsc8700000gn/T/pip-build-env-kxuy583w/overlay/lib/python3.8/site-packages/extension_helpers/_setup_helpers.py", line 67, in get_extensions ext_modules.extend(setuppkg.get_extensions()) File "./astropy/modeling/setup_package.py", line 59, in get_extensions from jinja2 import Environment, FileSystemLoader File "/private/var/folders/vx/5x_d29f50t7387_bmsjsc8700000gn/T/pip-build-env-kxuy583w/overlay/lib/python3.8/site-packages/jinja2/init.py", line 33, in from jinja2.environment import Environment, Template File "/private/var/folders/vx/5x_d29f50t7387_bmsjsc8700000gn/T/pip-build-env-kxuy583w/overlay/lib/python3.8/site-packages/jinja2/environment.py", line 15, in from jinja2 import nodes File "/private/var/folders/vx/5x_d29f50t7387_bmsjsc8700000gn/T/pip-build-env-kxuy583w/overlay/lib/python3.8/site-packages/jinja2/nodes.py", line 19, in from jinja2.utils import Markup File "/private/var/folders/vx/5x_d29f50t7387_bmsjsc8700000gn/T/pip-build-env-kxuy583w/overlay/lib/python3.8/site-packages/jinja2/utils.py", line 642, in from markupsafe import Markup, escape, soft_unicode ImportError: cannot import name 'soft_unicode' from 'markupsafe' (/private/var/folders/vx/5x_d29f50t7387_bmsjsc8700000gn/T/pip-build-env-kxuy583w/overlay/lib/python3.8/site-packages/markupsafe/init.py)

    ERROR: Command errored out with exit status 1: /Users/jardik/Documents/overlay/ML-auto-baseball-pitching-overlay/env-MLauto/bin/python3 /Users/jardik/Documents/overlay/ML-auto-baseball-pitching-overlay/env-MLauto/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /var/folders/vx/5x_d29f50t7387_bmsjsc8700000gn/T/tmp8cv3k60q Check the logs for full command output. WARNING: You are using pip version 19.2.3, however version 22.0.4 is available. You should consider upgrading via the 'pip install --upgrade pip' command. (env-MLauto) MacBook-Pro-2:ML-auto-baseball-pitching-overlay jardik$ `

    opened by MarekOstrihon 0
  • Hoping for Help

    Hoping for Help

    This is phenomenal work. However, I can't get it to work with some college baseball videos I've pulled and I was hoping for help. I've been using HD ESPN streams but I always get the "Error: Sorry we could not get enough baseball detection from the video, video output.mp4 will not be overlayed expected non-empty vector for x" issue.

    Wonder if you had any tips?

    opened by abrotherbear 0
  • Baseball Dataset

    Baseball Dataset

    Hey, cool project you got going on, I'm currently looking to also work on it, but was wondering if you are willing to share a labeled baseball dataset? At least I assume you manually created a dataset for the yolov4 model to fine-tune on, am I right? If you would rather not share it, then I understand that completely, just wondering!

    Kind regards

    opened by mtuit 0
Owner
Tony Chou
๐Ÿ’ป Just a student.
Tony Chou
Automatic self-diagnosis program (python required)Automatic self-diagnosis program (python required)

auto-self-checker ์ž๋™์œผ๋กœ ์ž๊ฐ€์ง„๋‹จ ํ•ด์ฃผ๋Š” ํ”„๋กœ๊ทธ๋žจ(python ํ•„์š”) ์ค‘์š” ์ด ํ”„๋กœ๊ทธ๋žจ์ด ์‹คํ–‰๋ ๋•Œ์—๋Š” ์ ˆ๋Œ€๋กœ ๋งˆ์šฐ์Šคํฌ์ธํ„ฐ๋ฅผ ์›€์ง์ด๊ฑฐ๋‚˜ ํ‚ค๋ณด๋“œ๋ฅผ ๊ฑด๋“œ๋ฆฌ๋ฉด ์•ˆ๋œ๋‹ค(ํ™”๋ฉด์ธ์‹, ๋งˆ์šฐ์Šคํฌ์ธํ„ฐ๋กœ ์ง์ ‘ ํด๋ฆญ) ์‚ฌ์šฉ๋ฒ• ํ”„๋กœ๊ทธ๋žจ์„ ๊ตฌ๋™ํ•  ํด๋” ๋‚ด์˜ cmd์ฐฝ์—์„œ pip

null 1 Dec 30, 2021
gitใ€ŠUSD-Seg:Learning Universal Shape Dictionary for Realtime Instance Segmentationใ€‹(2020) GitHub: [fig2]

USD-Seg This project is an implement of paper USD-Seg:Learning Universal Shape Dictionary for Realtime Instance Segmentation, based on FCOS detector f

Ruolin Ye 80 Nov 28, 2022
Frigate - NVR With Realtime Object Detection for IP Cameras

A complete and local NVR designed for HomeAssistant with AI object detection. Uses OpenCV and Tensorflow to perform realtime object detection locally for IP cameras.

Blake Blackshear 6.4k Dec 31, 2022
PyTorch Implementation of Realtime Multi-Person Pose Estimation project.

PyTorch Realtime Multi-Person Pose Estimation This is a pytorch version of Realtime_Multi-Person_Pose_Estimation, origin code is here Realtime_Multi-P

Dave Fang 157 Nov 12, 2022
Realtime Face Anti Spoofing with Face Detector based on Deep Learning using Tensorflow/Keras and OpenCV

Realtime Face Anti-Spoofing Detection ?? Realtime Face Anti Spoofing Detection with Face Detector to detect real and fake faces Please star this repo

Prem Kumar 86 Aug 3, 2022
Code repo for realtime multi-person pose estimation in CVPR'17 (Oral)

Realtime Multi-Person Pose Estimation By Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh. Introduction Code repo for winning 2016 MSCOCO Keypoints Cha

Zhe Cao 4.9k Dec 31, 2022
Tacotron 2 - PyTorch implementation with faster-than-realtime inference

Tacotron 2 (without wavenet) PyTorch implementation of Natural TTS Synthesis By Conditioning Wavenet On Mel Spectrogram Predictions. This implementati

NVIDIA Corporation 4.1k Jan 3, 2023
Sign Language is detected in realtime using video sequences. Our approach involves MediaPipe Holistic for keypoints extraction and LSTM Model for prediction.

RealTime Sign Language Detection using Action Recognition Approach Real-Time Sign Language is commonly predicted using models whose architecture consi

Rishikesh S 15 Aug 20, 2022
Dynamic Realtime Animation Control

Our project is targeted at making an application that dynamically detects the userโ€™s expressions and gestures and projects it onto an animation software which then renders a 2D/3D animation realtime that gets broadcasted live.

Harsh Avinash 10 Aug 1, 2022
RealTime Emotion Recognizer for Machine Learning Study Jam's demo

Emotion recognizer Table of contents Clone project Dataset Install dependencies Main program Demo 1. Clone project git clone https://github.com/GDSC20

Google Developer Student Club - UIT 1 Oct 5, 2021