This is an example of building a video Question-Answer system using Jina.

Related tags

example-video-qa
Overview

example-video-search

This is an example of building a video Question-Answer system using Jina.

The index data is subtitle files of YouTube videos. After indexing, you can query with questions in natural langurage and retrieve the related video together with the timestamp that the corresponding answer appears.

Prerequisites

pip install -r requirements.txt

Usage

By default, we index the subtitle file, toy-data/zvXkQkqd2I8.vtt

python app.py -m index

Query with questions,

python app.py -m query

To start the video UI, run the following codes and open http://localhost:3000/video/ in your browser.

git clone https://github.com/jina-ai/jina-ui.git
cd jina-ui
git checkout showcase-video-search
yarn install
yarn jinajs build
yarn showcases dev

How it works

The index flow is as below. The sentences are extracted from the subtitle file. The meta information of the sentences are stored in the LMDBStorage. In the other pathway, the sentences of the subtitles are encoded by the DPRTextEncoder and afterwards the result embeddings are stored in the SimpleIndexer.

The query flow is as shown below.

  1. The input query is a question which is encoded into embeddings by using DPRTextEncoder.
  2. The embedding of the query question is used to retrieve the sentences from SimpleIndexer.
  3. After retrieving the indexed sentences, we use MatchExpander to extend the questions so that the sentences from the neighboring timestamps are considered as candidate sentences as well.
  4. The candidate sentences retrieved from the above steps have no text or meta information. We retrieve the meta information from LMDBStorage.
  5. Rank the candidate sentences and extract the exact answers from the sentences by using DPRReaderRanker.
  6. Get the timestamp and video uri information about the answer candidates with Text2Frame

How to index my own data? [W.I.P.]

  1. download the subtitle files
youtube-dl --write-auto-sub --skip-download https://www.youtube.com/watch\?v\=zvXkQkqd2I8 -o data/zvXkQkqd2I8
  1. run the following
python app.py -m index -d data
python app.py -m query
Issues
Owner
Jina AI
A Neural Search Company. We provide the cloud-native neural search solution powered by state-of-the-art AI technology.
Jina AI
pygamevideo module helps developer to embed videos into their Pygame display

pygamevideo module helps developer to embed videos into their Pygame display. Audio playback doesn't use pygame.mixer.

Kadir Aksoy 7 Oct 10, 2021
High-performance cross-platform Video Processing Python framework powerpacked with unique trailblazing features :fire:

Releases | Gears | Documentation | Installation | License VidGear is a High-Performance Video Processing Python Library that provides an easy-to-use,

Abhishek Thakur 1.9k Oct 22, 2021
Python and OpenCV-based scene cut/transition detection program & library.

Video Scene Cut Detection and Analysis Tool Latest Release: v0.5.6.1 (October 11, 2021) Main Webpage: py.scenedetect.com Documentation: manual.scenede

Brandon Castellano 1.4k Oct 25, 2021
plumi video sharing

December 2017 update We are moving tickets from the Plumi tracker (trac.plumi.org) here, for historical reasons. Plumi video sharing system Plumi is a

Plumi 108 Sep 25, 2021
Your self hosted Youtube media server

The Tube Archivist Your self hosted Youtube media server Core functionality Subscribe to your favourite Youtube channels Download Videos using yt-dlp

Simon 506 Oct 24, 2021
MoviePy is a Python library for video editing, can read and write all the most common audio and video formats

MoviePy is a Python library for video editing: cutting, concatenations, title insertions, video compositing (a.k.a. non-linear editing), video processing, and creation of custom effects. See the gallery for some examples of use.

null 8.6k Oct 17, 2021
video streaming userbot (vsu) based on pytgcalls for streaming video trought the telegram video chat group.

VIDEO STREAM USERBOT ✨ an another telegram userbot for streaming video trought the telegram video chat. Environmental Variables ?? API_ID : Get this v

levina 6 Oct 17, 2021
OpenShot Video Editor is an award-winning free and open-source video editor for Linux, Mac, and Windows, and is dedicated to delivering high quality video editing and animation solutions to the world.

OpenShot Video Editor is an award-winning free and open-source video editor for Linux, Mac, and Windows, and is dedicated to delivering high quality v

OpenShot Studios, LLC 2.4k Oct 24, 2021
Video processing routines for SciPy

scikit-video Video Processing SciKit BETA Video processing algorithms, including I/O, quality metrics, temporal filtering, motion/object detection, mo

Alex Izvorski 105 Aug 18, 2021
Youtube as covert-channel - Control systems remotely and execute commands by uploading videos to Youtube

covert-tube A program to control systems remotely by uploading videos to Youtube using Python to create the videos and the listener, emulating some ma

Ricardo Ruiz 65 Oct 20, 2021
Python application that can be used to generate video thumbnail for mp4 and mkv file types.

Thumbnail Generator ?? What is This This is a Python application that can be used to generate video thumbnail for mp4 and mkv file types. Installation

Tharindu N. 6 Jul 26, 2021
A python program which converts images and video into excel spreadsheets.

image2excel A program which converts images and video into Excel spreadsheets. Usage examples can be found in examples Videos can take a long time to

Oscar Peace 2 Aug 9, 2021
Repository to create Ascii art in CMD based on video file.

Made to take any file format, and transform it into ascii art, displayed as a video in the cmd. If the cmd formatting is wrong, try zooming a little and remember to make cmd fullscreen. I made my cmd fullscreen, and zoomed out one tic. Written in Python 3.9

null 51 Aug 16, 2021
A youtube video link or id to video thumbnail python package.

Youtube-Video-Thumbnail A youtube video link or id to video thumbnail python package. Made with Python3

Fayas Noushad 7 Oct 18, 2021
Convert lecture videos to slides in one line. Takes an input of a directory containing your lecture videos and outputs a directory containing .PDF files containing the slides of each lecture.

Convert lecture videos to slides in one line. Takes an input of a directory containing your lecture videos and outputs a directory containing .PDF files containing the slides of each lecture.

Sidharth Anand 6 Oct 22, 2021
Cross-platform command-line AV1 / VP9 / HEVC / H264 encoding framework with per scene quality encoding

Av1an A cross-platform framework to streamline encoding Easy, Fast, Efficient and Feature Rich An easy way to start using AV1 / HEVC / H264 / VP9 / VP

Zen 473 Oct 15, 2021
TkVideoplayer - This is a simple library to play video files in tkinter.

TkVideoplayer - This is a simple library to play video files in tkinter.

Art/Paul 3 Oct 20, 2021
This program is to make a video based on Deep Dream

This program is to make a video based on Deep Dream. The program is modified from DeepDreamAnim and DeepDreamVideo with additional functions for bleding two frames based on the optical flows. It also supports the image division to apply the Deep Dream algorithm to a large image.

Aertist 19 Apr 20, 2021
Rembg Video Virtual Green Screen Edition

Rembg Video Virtual Green Screen Edition

Tim Scarfe 146 Oct 13, 2021