Using VideoBERT to tackle video prediction

Overview

VideoBERT

This repo reproduces the results of VideoBERT (https://arxiv.org/pdf/1904.01766.pdf). Inspiration was taken from https://github.com/MDSKUL/MasterProject, but this repo tackles video prediction rather than captioning and masked language modeling. On a side note, since this model is extremely small, the results that are displayed here are extremely basic. Feel free to increase the model size per your computational resources and change the inference file to include temperature if necessary (As of now I have not implemented temperature). Here are all the steps taken:

Step 1: Download 47k videos from the HowTo100M dataset

Using the HowTo100M dataset https://www.di.ens.fr/willow/research/howto100m/, filter out the cooking videos and download them for feature extraction. The dataset is also used for extracting images for each feature vector. The ids for the videos are contained in the ids.txt file.

Step 2: Do feature extraction with the I3D model

The I3D model is used to extract the features for every 1.5 seconds of video while saving the median image of the 1.5 seconds of video as well. I3D model used: https://tfhub.dev/deepmind/i3d-kinetics-600/1. Note that CUDA should be used to decrease the runtime. Here is the usage for the code to run:

$ python3 VideoBERT/VideoBERT/I3D/batch_extract.py -h
usage: batch_extract.py [-h] -f FILE_LIST_PATH -r ROOT_VIDEO_PATH -s FEATURES_SAVE_PATH -i IMGS_SAVE_PATH

optional arguments:
  -h, --help            show this help message and exit
  -f FILE_LIST_PATH, --file-list-path FILE_LIST_PATH
                        path to file containing video file names
  -r ROOT_VIDEO_PATH, --root-video-path ROOT_VIDEO_PATH
                        root directory containing video files
  -s FEATURES_SAVE_PATH, --features-save-path FEATURES_SAVE_PATH
                        directory in which to save features
  -i IMGS_SAVE_PATH, --imgs-save-path IMGS_SAVE_PATH
                        directory in which to save images

Step 3: Hierarchical Minibatch K-means

To find the centroids for the feature vectors, minibatch k-means is used hierarchically to save time and memory. After this, the nearest feature vector for each centroid is found, and the corresponding image is chosen to represent tht centroid. To use the hierarchical minibatch k-means independently for another project, consider using the python package hkmeans-minibatch, which is also used in this VideoBERT project (https://github.com/ammesatyajit/hierarchical-minibatch-kmeans).

Here is the usage for the kmeans code:

$ python3 VideoBERT/VideoBERT/I3D/minibatch_hkmeans.py -h 
usage: minibatch_hkmeans.py [-h] -r ROOT_FEATURE_PATH -p FEATURES_PREFIX [-b BATCH_SIZE] -s SAVE_DIR -c CENTROID_DIR

optional arguments:
  -h, --help            show this help message and exit
  -r ROOT_FEATURE_PATH, --root-feature_path ROOT_FEATURE_PATH
                        path to folder containing all the video folders with the features
  -p FEATURES_PREFIX, --features-prefix FEATURES_PREFIX
                        prefix that is common between the desired files to read
  -b BATCH_SIZE, --batch-size BATCH_SIZE
                        batch_size to use for the minibatch kmeans
  -s SAVE_DIR, --save-dir SAVE_DIR
                        save directory for hierarchical kmeans vectors
  -c CENTROID_DIR, --centroid-dir CENTROID_DIR
                        directory to save the centroids in

Note that after this step the centroids will need to be concatenated for ease of use.

After doing kmeans, the image representing each centroid needs to be found to display the video during inference.

$ python3 VideoBERT/VideoBERT/data/centroid_to_img.py -h 
usage: centroid_to_img.py [-h] -f ROOT_FEATURES -i ROOT_IMGS -c CENTROID_FILE -s SAVE_FILE

optional arguments:
  -h, --help            show this help message and exit
  -f ROOT_FEATURES, --root-features ROOT_FEATURES
                        path to folder containing all the video folders with the features
  -i ROOT_IMGS, --root-imgs ROOT_IMGS
                        path to folder containing all the video folders with the images corresponding to the features
  -c CENTROID_FILE, --centroid-file CENTROID_FILE
                        the .npy file containing all the centroids
  -s SAVE_FILE, --save-file SAVE_FILE
                        json file to save the centroid to image dictionary in

Step 4: Label and group data

Using the centroids, videos are tokenized and text captions are punctuated. Using the timestamps for each caption, video ids are extracted and paired with the text captions in the training data file. Captions can be found here: https://www.rocq.inria.fr/cluster-willow/amiech/howto100m/.

The python file below tokenizes the videos:

$ python3 VideoBERT/VideoBERT/data/label_data.py -h     
usage: label_data.py [-h] -f ROOT_FEATURES -c CENTROID_FILE -s SAVE_FILE

optional arguments:
  -h, --help            show this help message and exit
  -f ROOT_FEATURES, --root-features ROOT_FEATURES
                        path to folder containing all the video folders with the features
  -c CENTROID_FILE, --centroid-file CENTROID_FILE
                        the .npy file containing all the centroids
  -s SAVE_FILE, --save-file SAVE_FILE
                        json file to save the labelled data to

After that the following file can be run to both punctuate text and group the text with the corresponding video. This uses the Punctuator module, which requires a .pcl model file to punctuate the data.

$ python3 VideoBERT/VideoBERT/data/punctuate_text.py -h 
usage: punctuate_text.py [-h] -c CAPTIONS_PATH -p PUNCTUATOR_MODEL -l LABELLED_DATA -f ROOT_FEATURES -s SAVE_PATH

optional arguments:
  -h, --help            show this help message and exit
  -c CAPTIONS_PATH, --captions-path CAPTIONS_PATH
                        path to filtered captions
  -p PUNCTUATOR_MODEL, --punctuator-model PUNCTUATOR_MODEL
                        path to punctuator .pcl model
  -l LABELLED_DATA, --labelled-data LABELLED_DATA
                        path to labelled data json file
  -f ROOT_FEATURES, --root-features ROOT_FEATURES
                        directory with all the video features
  -s SAVE_PATH, --save-path SAVE_PATH
                        json file to save training data to

If desired, an evaluation data file can be created by splitting the training data file.

Step 5: Training

The training data from before is used to train a next token prediction transformer. The saved model and tokenizer is used for inference in the next step. here is the usage of the train.py file.

$ python3 VideoBERT/VideoBERT/train/train.py -h
usage: train.py [-h] --output_dir OUTPUT_DIR [--should_continue] [--model_name_or_path MODEL_NAME_OR_PATH] [--train_data_path TRAIN_DATA_PATH] [--eval_data_path EVAL_DATA_PATH] [--config_name CONFIG_NAME] [--block_size BLOCK_SIZE]
                [--per_gpu_train_batch_size PER_GPU_TRAIN_BATCH_SIZE] [--gradient_accumulation_steps GRADIENT_ACCUMULATION_STEPS] [--learning_rate LEARNING_RATE] [--weight_decay WEIGHT_DECAY] [--adam_epsilon ADAM_EPSILON]
                [--max_grad_norm MAX_GRAD_NORM] [--num_train_epochs NUM_TRAIN_EPOCHS] [--max_steps MAX_STEPS] [--log_dir LOG_DIR] [--warmup_steps WARMUP_STEPS] [--local_rank LOCAL_RANK] [--logging_steps LOGGING_STEPS]
                [--save_steps SAVE_STEPS] [--save_total_limit SAVE_TOTAL_LIMIT] [--overwrite_output_dir] [--overwrite_cache] [--seed SEED]

optional arguments:
  -h, --help            show this help message and exit
  --output_dir OUTPUT_DIR
                        The output directory where the model predictions and checkpoints will be written.
  --should_continue     Whether to continue from latest checkpoint in output_dir
  --model_name_or_path MODEL_NAME_OR_PATH
                        The model checkpoint for weights initialization. Leave None if you want to train a model from scratch.
  --train_data_path TRAIN_DATA_PATH
                        The json file for training the model
  --eval_data_path EVAL_DATA_PATH
                        The json file for evaluating the model
  --config_name CONFIG_NAME
                        Optional pretrained config name or path if not the same as model_name_or_path. If both are None, initialize a new config.
  --block_size BLOCK_SIZE
                        Optional input sequence length after tokenization.The training dataset will be truncated in block of this size for training.Default to the model max input length for single sentence inputs (take into account
                        special tokens).
  --per_gpu_train_batch_size PER_GPU_TRAIN_BATCH_SIZE
                        Batch size per GPU/CPU for training.
  --gradient_accumulation_steps GRADIENT_ACCUMULATION_STEPS
                        Number of updates steps to accumulate before performing a backward/update pass.
  --learning_rate LEARNING_RATE
                        The initial learning rate for Adam.
  --weight_decay WEIGHT_DECAY
                        Weight decay if we apply some.
  --adam_epsilon ADAM_EPSILON
                        Epsilon for Adam optimizer.
  --max_grad_norm MAX_GRAD_NORM
                        Max gradient norm.
  --num_train_epochs NUM_TRAIN_EPOCHS
                        Total number of training epochs to perform.
  --max_steps MAX_STEPS
                        If > 0: set total number of training steps to perform. Override num_train_epochs.
  --log_dir LOG_DIR     Directory to store the logs.
  --warmup_steps WARMUP_STEPS
                        Linear warmup over warmup_steps.
  --local_rank LOCAL_RANK
                        For distributed training: local_rank
  --logging_steps LOGGING_STEPS
                        Log every X updates steps.
  --save_steps SAVE_STEPS
                        Save checkpoint every X updates steps.
  --save_total_limit SAVE_TOTAL_LIMIT
                        Limit the total amount of checkpoints, delete the older checkpoints in the output_dir, does not delete by default
  --overwrite_output_dir
                        Overwrite the content of the output directory
  --overwrite_cache     Overwrite the cached training and evaluation sets
  --seed SEED           random seed for initialization

Step 6: Inference

Model is used for predicting video sequences and results can be seen visually. Note that since the model does uses vector quantized images as tokens, it only understands the actions and approximate background of the scene, not the exact person or dish. Here are some samples:

out1 out2 out3 out4 out5

Here is the usage for the inference file. Feel free to modify it to suit any specific needs:

$ python3 VideoBERT/VideoBERT/evaluation/inference.py -h 
usage: inference.py [-h] [--model_name_or_path MODEL_NAME_OR_PATH] --output_dir OUTPUT_DIR [--example_id EXAMPLE_ID] [--seed SEED]

optional arguments:
  -h, --help            show this help message and exit
  --model_name_or_path MODEL_NAME_OR_PATH
                        The model checkpoint for weights initialization. Leave None if you want to train a model from scratch.
  --output_dir OUTPUT_DIR
                        The output directory where the checkpoint is.
  --example_id EXAMPLE_ID
                        The index of the eval set for evaluating the model
  --seed SEED           random seed for initialization
Comments
  • minibatch package error

    minibatch package error

    Thanks for your work in implementing video bert! I was receiving the following error when playing around with the code, training on a set of 300 videos with a batch size of 12:

    Traceback (most recent call last): File "VideoBERT/VideoBERT/I3D/minibatch_hkmeans.py", line 30, in main() File "VideoBERT/VideoBERT/I3D/minibatch_hkmeans.py", line 26, in main hkmeans(root, prefix, 4, 12, batch_size, 15, save_dir, 'vecs', centroid_dir) File "/opt/conda/envs/videobert/lib/python3.7/site-packages/hkmeans_minibatch-1.0.2-py3.7.egg/hkmeans_minibatch/hkmeans.py", line 99, in hkmeans hkmeans_recursive(root, prefix, h, k, batch_size, epochs, save_dir, save_prefix, centroid_dir) File "/opt/conda/envs/videobert/lib/python3.7/site-packages/hkmeans_minibatch-1.0.2-py3.7.egg/hkmeans_minibatch/hkmeans.py", line 91, in hkmeans_recursive save_prefix.format(i), centroid_dir, cur_h=cur_h + 1) File "/opt/conda/envs/videobert/lib/python3.7/site-packages/hkmeans_minibatch-1.0.2-py3.7.egg/hkmeans_minibatch/hkmeans.py", line 87, in hkmeans_recursive save_sorted_vectors(centroids, labelled_data, batch_size, save_dir, save_prefix) File "/opt/conda/envs/videobert/lib/python3.7/site-packages/hkmeans_minibatch-1.0.2-py3.7.egg/hkmeans_minibatch/hkmeans.py", line 56, in save_sorted_vectors sorted_vecs.append(np.expand_dims(vectors[j], axis=0)) IndexError: index 12 is out of bounds for axis 0 with size 12

    Is this an issue with dataset size?

    opened by schko 1
  • ValueError: Incorrect number of features. Got 600 features, expected 12

    ValueError: Incorrect number of features. Got 600 features, expected 12

    Traceback (most recent call last): File "VideoBERT/data/label_data.py", line 32, in data_dict[folder].extend(kmeans.predict(np.load(os.path.join(features_root, folder, features)))) File "Python\Python38\lib\site-packages\sklearn\cluster_kmeans.py", line 1915, in predict X = self._check_test_data(X) File "Python\Python38\lib\site-packages\sklearn\cluster_kmeans.py", line 998, in _check_test_data raise ValueError( ValueError: Incorrect number of features. Got 600 features, expected 12.

    I get this error when I try to run the 'label_data.py' file. I followed the previous steps twice just to make sure I didn't mess up anything but I still get to this error. Maybe it is something small that I am missing but I am not sure. Could someone provide assistance?

    opened by Uvir127 0
  • meaning of the README.md

    meaning of the README.md

    Could you please explain the meaning of "change the inference file to include temperature if necessary" in README.md?And I am confused what should I do if I want to use a pre-trained videobert module?

    opened by changjiangamy 0
  • need this files

    need this files

    root_path = '/content/drive/My Drive/VideoBERT' data_path = '/content/drive/My Drive/VideoBERT/training_data.csv' centers_file = '/content/drive/My Drive/VideoBERT/centers.npy' val_youcook = '/content/drive/My Drive/VideoBERT/val_data.csv'

    opened by harshraj32 0
  • AttributeError: 'MiniBatchKMeans' object has no attribute 'cluster_centers_'

    AttributeError: 'MiniBatchKMeans' object has no attribute 'cluster_centers_'

    Traceback (most recent call last): File "VideoBERT/VideoBERT/I3D/minibatch_hkmeans.py", line 30, in main() File "VideoBERT/VideoBERT/I3D/minibatch_hkmeans.py", line 26, in main hkmeans(root, prefix, 4, 12, batch_size, 15, save_dir, 'vecs', centroid_dir) File "/usr/local/lib/python3.7/dist-packages/hkmeans_minibatch/hkmeans.py", line 99, in hkmeans hkmeans_recursive(root, prefix, h, k, batch_size, epochs, save_dir, save_prefix, centroid_dir) File "/usr/local/lib/python3.7/dist-packages/hkmeans_minibatch/hkmeans.py", line 85, in hkmeans_recursive centroids, labelled_data = minibatch_kmeans(root, prefix, k, batch_size, epochs) File "/usr/local/lib/python3.7/dist-packages/hkmeans_minibatch/hkmeans.py", line 42, in minibatch_kmeans return kmeans.cluster_centers_, labelled_data AttributeError: 'MiniBatchKMeans' object has no attribute 'cluster_centers_'

    the line i ran without any changes : !python3 VideoBERT/VideoBERT/I3D/minibatch_hkmeans.py -r /content/VideoBERT/VideoBERT/data/features -b 1 -s /content/VideoBERT/VideoBERT/data/kmean_vectors -c /content/VideoBERT/VideoBERT/data/centroids -p hkm

    opened by harshraj32 4
  • Questions to understand this repo well :D

    Questions to understand this repo well :D

    First, thank you for your great work :D

    Question is like the title asked.

    Using the centroids, videos are tokenized and text captions are punctuated. Using the timestamps for each caption, video ids are extracted and paired with the text captions in the training data file. Captions can be found here: https://www.rocq.inria.fr/cluster-willow/amiech/howto100m/.

    In short, which file should I download if I want to match the 'Captions'(appears in the last sentence of quotes) ? I have downloaded the howto100m_captions.zip(2.3G), is it correct ? I saw the files in it are all .csv file ;_; What's more, Has anyone run the repo which the author said that he/she inspired by ? the repo is https://github.com/MDSKUL/MasterProject

    opened by FormerAutumn 16
Owner
null
Video-Captioning - A machine Learning project to generate captions for video frames indicating the relationship between the objects in the video

Video-Captioning - A machine Learning project to generate captions for video frames indicating the relationship between the objects in the video

null 1 Jan 23, 2022
Price-Prediction-For-a-Dream-Home - A machine learning based linear regression trained model for house price prediction.

Price-Prediction-For-a-Dream-Home ROADMAP TO THIS LINEAR REGRESSION BASED HOUSE PRICE PREDICTION PREDICTION MODEL Import all the dependencies of the p

DIKSHA DESWAL 1 Dec 29, 2021
Doge-Prediction - Coding Club prediction ig

Doge-Prediction Coding Club prediction ig Basically: Create an application that

null 1 Jan 10, 2022
Sign Language is detected in realtime using video sequences. Our approach involves MediaPipe Holistic for keypoints extraction and LSTM Model for prediction.

RealTime Sign Language Detection using Action Recognition Approach Real-Time Sign Language is commonly predicted using models whose architecture consi

Rishikesh S 15 Aug 20, 2022
Video lie detector using xgboost - A video lie detector using OpenFace and xgboost

video_lie_detector_using_xgboost a video lie detector using OpenFace and xgboost

null 2 Jan 11, 2022
VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learning

This is a release of our VIMPAC paper to illustrate the implementations. The pretrained checkpoints and scripts will be soon open-sourced in HuggingFace transformers.

Hao Tan 74 Dec 3, 2022
Implementation of FitVid video prediction model in JAX/Flax.

FitVid Video Prediction Model Implementation of FitVid video prediction model in JAX/Flax. If you find this code useful, please cite it in your paper:

Google Research 62 Nov 25, 2022
MAU: A Motion-Aware Unit for Video Prediction and Beyond, NeurIPS2021

MAU (NeurIPS2021) Zheng Chang, Xinfeng Zhang, Shanshe Wang, Siwei Ma, Yan Ye, Xinguang Xiang, Wen GAo. Official PyTorch Code for "MAU: A Motion-Aware

ZhengChang 20 Nov 25, 2022
[CVPR 2022] Official PyTorch Implementation for "Reference-based Video Super-Resolution Using Multi-Camera Video Triplets"

Reference-based Video Super-Resolution (RefVSR) Official PyTorch Implementation of the CVPR 2022 Paper Project | arXiv | RealMCVSR Dataset This repo c

Junyong Lee 151 Dec 30, 2022
[NeurIPS 2020] Blind Video Temporal Consistency via Deep Video Prior

pytorch-deep-video-prior (DVP) Official PyTorch implementation for NeurIPS 2020 paper: Blind Video Temporal Consistency via Deep Video Prior TensorFlo

Yazhou XING 90 Oct 19, 2022
Pytorch implementation of our method for high-resolution (e.g. 2048x1024) photorealistic video-to-video translation.

vid2vid Project | YouTube(short) | YouTube(full) | arXiv | Paper(full) Pytorch implementation for high-resolution (e.g., 2048x1024) photorealistic vid

NVIDIA Corporation 8.1k Jan 1, 2023
Search Youtube Video and Get Video info

PyYouTube Get Video Data from YouTube link Installation pip install PyYouTube How to use it ? Get Videos Data from pyyoutube import Data yt = Data("ht

lokaman chendekar 35 Nov 25, 2022
We present a framework for training multi-modal deep learning models on unlabelled video data by forcing the network to learn invariances to transformations applied to both the audio and video streams.

Multi-Modal Self-Supervision using GDT and StiCa This is an official pytorch implementation of papers: Multi-modal Self-Supervision from Generalized D

Facebook Research 42 Dec 9, 2022
PySlowFast: video understanding codebase from FAIR for reproducing state-of-the-art video models.

PySlowFast PySlowFast is an open source video understanding codebase from FAIR that provides state-of-the-art video classification models with efficie

Meta Research 5.3k Jan 3, 2023
Eff video representation - Efficient video representation through neural fields

Neural Residual Flow Fields for Efficient Video Representations 1. Download MPI

null 41 Jan 6, 2023
Video-face-extractor - Video face extractor with Python

Python face extractor Setup Create the srcvideos and faces directories Put your

null 2 Feb 3, 2022
Age and Gender prediction using Keras

cnn_age_gender Age and Gender prediction using Keras Dataset example : Description : UTKFace dataset is a large-scale face dataset with long age span

XN3UR0N 58 May 3, 2022
OHLC Average Prediction of Apple Inc. Using LSTM Recurrent Neural Network

Stock Price Prediction of Apple Inc. Using Recurrent Neural Network OHLC Average Prediction of Apple Inc. Using LSTM Recurrent Neural Network Dataset:

Nouroz Rahman 410 Jan 5, 2023
Code and datasets for the paper "Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction" (RA-L, 2021)

Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction This is the code for the paper Combining E

Robotics and Perception Group 69 Dec 26, 2022