"Video Moment Retrieval from Text Queries via Single Frame Annotation" in SIGIR 2022.

Overview

ViGA: Video moment retrieval via Glance Annotation

This is the official repository of the paper "Video Moment Retrieval from Text Queries via Single Frame Annotation" published in SIGIR 2022.

https://arxiv.org/abs/2204.09409

Dependencies

This project has been tested on the following conda environment.

$ conda create --name viga python=3.7
$ source activate viga
(viga)$ conda install pytorch=1.10.0 cudatoolkit=11.3.1
(viga)$ pip install numpy scipy pyyaml tqdm 

Data preparation

This repository contains our glance annotations already. To replicate our work, one should prepare extra data and finally get the following structure.

ckpt/                                 our pre-trained model, available at https://drive.google.com/file/d/1S4e8XmIpiVFJKSSJ4Tig4qN0yaCwiVLs/view?usp=sharing
data/
+-- activitynetcaptions/
|   +-- c3d/                    
|   +-- annotations/
|   |   +-- glance/
|   |   |   +-- train.json                
|   |   |   +-- val_1.json                
|   |   |   +-- val_2.json   
|   |   +-- train.json                downloaded
|   |   +-- val_1.json                downloaded
|   |   +-- val_2.json                downloaded
+-- charadessta/
|   +-- i3d/                     
|   +-- c3d/ 
|   +-- vgg/
|   +-- annotations/
|   |   +-- glance/
|   |   |   +-- charades_sta_train.txt
|   |   |   +-- charades_sta_test.txt
|   |   +-- charades_sta_train.txt    downloaded
|   |   +-- charades_sta_test.txt     downloaded
|   |   +-- Charades_v1_train.csv     downloaded
|   |   +-- Charades_v1_test.csv      downloaded
+-- tacos/
|   +-- c3d/ 
|   +-- annotations/
|   |   +-- glance/
|   |   |   +-- train.json                
|   |   |   +-- test.json                 
|   |   |   +-- val.json
|   |   +-- train.json                downloaded
|   |   +-- test.json                 downloaded
|   |   +-- val.json                  downloaded
glove.840B.300d.txt                   downloaded from https://nlp.stanford.edu/data/glove.840B.300d.zip

1. ActivityNet Captions

c3d feature

Downloaded from http://activity-net.org/challenges/2016/download.html. We extracted the features from sub_activitynet_v1-3.c3d.hdf5 as individual files.

Folder contains 19994 vid.npys, each of shape (T, 500).

annotation

Downloaded from https://cs.stanford.edu/people/ranjaykrishna/densevid/

2. Charades-STA

c3d feature

We extracted this by ourselves, due to the lack of storage resource we are currently not able to make this feature publicly available.

Folder contains 9848 vid.npys, each of shape (T, 4096).

i3d feature

Downloaded from https://github.com/JonghwanMun/LGI4temporalgrounding. This is the features extracted from I3D (finetuned on Charades). We processed them by trimming off unnecessary dimensions.

Folder contains 9848 vid.npys, each of shape (T, 1024).

vgg feature

Downloaded from https://github.com/microsoft/2D-TAN. We processed the data by converting the downloaded version vgg_rgb_features.hdf5 into numpy arrays.

Folder contains 6672 vid.npys, each of shape (T, 4096).

annotation

Downloaded from https://github.com/jiyanggao/TALL

3. TACoS

c3dfeature

Downloaded from https://github.com/microsoft/2D-TAN. We extracted the features from tall_c3d_features.hdf5 as individual files.

Folder contains 127 vid.npys, each of shape (T, 4096).

annotation

Downloaded from https://github.com/microsoft/2D-TAN

2. Run

Our models were trained using the following commands.

(viga)$ CUDA_VISIBLE_DEVICES=0 python -m src.experiment.train --task activitynetcaptions
(viga)$ CUDA_VISIBLE_DEVICES=0 python -m src.experiment.train --task charadessta
(viga)$ CUDA_VISIBLE_DEVICES=0 python -m src.experiment.train --task tacos

Our trained models were evaluated using the following commands.

(viga)$ CUDA_VISIBLE_DEVICES=0 python -m src.experiment.eval --exp ckpt/activitynetcaptions
(viga)$ CUDA_VISIBLE_DEVICES=0 python -m src.experiment.eval --exp ckpt/charadessta_c3d
(viga)$ CUDA_VISIBLE_DEVICES=0 python -m src.experiment.eval --exp ckpt/charadessta_i3d
(viga)$ CUDA_VISIBLE_DEVICES=0 python -m src.experiment.eval --exp ckpt/charadessta_vgg
(viga)$ CUDA_VISIBLE_DEVICES=0 python -m src.experiment.eval --exp ckpt/tacos
Issues
  • errors occurs when it comes to run the Charades-STA dataset.

    errors occurs when it comes to run the Charades-STA dataset.

    Hi, thanks for your wonderful work , however, I have met the following error when I run the Charades-STA dataset (everything goes well when I run the other two datasets, tacos and activity net). Would you mind helping figure it out? Thanks.

    errors log

    Training epoch 1 with lr 0.0001: 0%| | 0/24 [00:00<?, ?it/s] Training epoch 1 with lr 0.0001: 0%| | 0/24 [00:01<?, ?it/s] Traceback (most recent call last): File "/data1/zhekun/anaconda3/envs/viga/lib/python3.7/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/data1/zhekun/anaconda3/envs/viga/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/data1/chentongbao/VCMR/ViGA/src/experiment/train.py", line 90, in train(config) File "/data1/chentongbao/VCMR/ViGA/src/experiment/train.py", line 56, in train desc="Training epoch {} with lr {}".format(epoch, model.optimizer.param_groups[0]["lr"]) File "/data1/zhekun/anaconda3/envs/viga/lib/python3.7/site-packages/tqdm/std.py", line 1195, in iter for obj in iterable: File "/data1/zhekun/anaconda3/envs/viga/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 521, in next data = self._next_data() File "/data1/zhekun/anaconda3/envs/viga/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data return self._process_data(data) File "/data1/zhekun/anaconda3/envs/viga/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data data.reraise() File "/data1/zhekun/anaconda3/envs/viga/lib/python3.7/site-packages/torch/_utils.py", line 434, in reraise raise exception RuntimeError: Caught RuntimeError in DataLoader worker process 0. Original Traceback (most recent call last): File "/data1/zhekun/anaconda3/envs/viga/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop data = fetcher.fetch(index) File "/data1/zhekun/anaconda3/envs/viga/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch return self.collate_fn(data) File "/data1/chentongbao/VCMR/ViGA/src/dataset/dataset.py", line 116, in collate_fn batch[k] = torch.stack(batch[k], dim=0) RuntimeError: stack expects each tensor to be equal size, but got [87, 1, 42, 1024] at entry 0 and [72, 1, 57, 1024] at entry 1

    opened by tongbaochen 3
Owner
Ran Cui
Ran Cui
QHack-2022 - Solutions to the Coding Challenges of QHack 2022

QHack 2022 Problems from Coding Challenges 2022. Rules and how it works To test

Isacco Gobbi 1 Feb 14, 2022
CVE-2022-22536 - SAP memory pipes(MPI) desynchronization vulnerability CVE-2022-22536

CVE-2022-22536 SAP memory pipes desynchronization vulnerability(MPI) CVE-2022-22

antx 46 Jun 18, 2022
Cve-2022-23131 - Cve-2022-23131 zabbix-saml-bypass-exp

cve-2022-23131 cve-2022-23131 zabbix-saml-bypass-exp replace [zbx_signed_session

东方有鱼名为咸 97 Jun 19, 2022
Windows Server 2016, 2019, 2022 Extracter & Recovery

Parsing files from Deduplicated volumes. It can also recover deleted files from NTFS Filesystem that were deduplicated. Installation git clone https:/

null 1 Dec 27, 2021
Proof of concept of CVE-2022-21907 Double Free in http.sys driver, triggering a kernel crash on IIS servers

CVE-2022-21907 - Double Free in http.sys driver Summary An unauthenticated attacker can send an HTTP request with an "Accept-Encoding" HTTP request he

Podalirius 66 Jun 28, 2022
This repository detects a system vulnerable to CVE-2022-21907 and protects against this vulnerability if desired

This repository detects a system vulnerable to CVE-2022-21907 and protects against this vulnerability if desired

null 22 Jun 6, 2022
CVE-2022-21907 Vulnerability PoC

CVE-2022-21907 Description POC for CVE-2022-21907: HTTP Protocol Stack Remote Code Execution Vulnerability. create by antx at 2022-01-17, just some sm

Michele 6 Apr 15, 2022
AttractionFinder - 2022 State Qualified FBLA Attraction Finder Application

Attraction Finder Developers: Riyon Praveen, Aaron Bijoy, & Yash Vora How It Wor

$ky 2 Feb 9, 2022
Valeria stealer- - (4Feb 2022) program detects wifi saved passwords in your ROM

Valeria_stealer- Requirements : python 3.9.2 and higher (4Feb 2022) program dete

Mikhail Yolkin 3 May 5, 2022
A Python script that can be used to check if a SAP system is affected by CVE-2022-22536

Vulnerability assessment for CVE-2022-22536 This repository contains a Python script that can be used to check if a SAP system is affected by CVE-2022

Onapsis Inc. 35 Jun 28, 2022
CVE-2022-23046 - SQL Injection Vulnerability on PhpIPAM v1.4.4

CVE-2022-23046 PhpIPAM v1.4.4 allows an authenticated admin user to inject SQL s

null 2 Feb 15, 2022
2022-bridge - Example code belonging to the Bridge pattern video

Let's Take The Bridge Pattern To The Next Level This video covers how the bridge

null 11 Jun 14, 2022
spring-cloud-gateway-rce CVE-2022-22947

Spring Cloud Gateway Actuator API SpEL表达式注入命令执行(CVE-2022-22947) 1.installation pip3 install -r requirements.txt 2.Usage $ python3 spring-cloud-gateway

k3rwin 7 Apr 8, 2022
Spring-0day/CVE-2022-22965

CVE-2022-22965 Spring Framework/CVE-2022-22965 Vulnerability ID: CVE-2022-22965/CNVD-2022-23942/QVD-2022-1691 Reproduce the vulnerability docker pull

iak 4 Apr 5, 2022
Source code for "A Two-Stream AMR-enhanced Model for Document-level Event Argument Extraction" @ NAACL 2022

TSAR Source code for NAACL 2022 paper: A Two-Stream AMR-enhanced Model for Document-level Event Argument Extraction. ?? Introduction We focus on extra

null 15 Jun 21, 2022
CVE-2022-22965 - CVE-2010-1622 redux

CVE-2022-22965 - vulnerable app and PoC Trial & error $ docker rm -f rce; docker build -t rce:latest . && docker run -d -p 8080:8080 --name rce rce:la

Duarte Duarte 20 Apr 12, 2022
CVE-2022-22965 : about spring core rce

CVE-2022-22965: Spring-Core-Rce EXP 特性: 漏洞探测(不写入 webshell,简单字符串输出) 自定义写入 webshell 文件名称及路径 不会追加写入到同一文件中,每次检测写入到不同名称 webshell 文件 支持写入 冰蝎 webshell 代理支持,可

东方有鱼名为咸 50 Jun 6, 2022
HTTP Protocol Stack Remote Code Execution Vulnerability CVE-2022-21907

CVE-2022-21907 Description POC for CVE-2022-21907: HTTP Protocol Stack Remote Code Execution Vulnerability. create by antx at 2022-01-17. Detail HTTP

赛欧思网络安全研究实验室 362 Jun 15, 2022