Cross-category Video Highlight Detection via Set-based Learning
Introduction
This project is an implementation of ``Cross-category Video Highlight Detection via Set-based Learning'' in PyTorch, which is accepted by ICCV 2021. We provide the codes for our proposed SL-module, DL-VHD and also some domain adaptation baselines in this repository.
More details of this work can be found in our paper: [Paper (arXiv)].
Prerequisites
We develop this project with Python3.6 and following Python packages:
Pytorch 1.5.0
cv2 3.4.2
einops 0.3.0
P.S. In our project, these packages can be successfully installed and work together under CUDA/9.0 and cuDNN/7.0.5.
Dataset and Pre-trained Model
YouTube Highlights dataset. You can download the dataset in this repository and put it under the path you like, e.g. ~/data/YouTube_Highlights/.
For training highlight detection models, you can convert the original videos in YouTube Highlights to video segments by following command:
python ./dataloaders/youtube_highlights_set.py
ActivityNet dataset. Because of the large size of the full dataset, we suggest you to crawl videos using this tool. We provide the names of the videos used in our work in this file. You can put the crawled videos under the path you like, e.g. ~/data/ActivityNet/.
To fit the format of highlight detection, you can convert the raw videos and annotations by following commands:
cd ./data_process
python process_ActivityNet.py
Pre-trained model. You can download the pre-trained C3D model in this link and put it under the path you like, e.g. ~/pretrained_models/.
Category-specific Video Highlight Detection
SL-module. To train and finally evaluate the SL-module, simply run:
python train_SL_module.py --gpu_id $device_id$ --src_category $cls$ \
--tgt_category $cls$ --use_transformer
Cross-category Video Highlight Detection
Source-only. To train and finally evaluate the Source-only model with SL-module, simply run:
python train_SL_module.py --gpu_id $device_id$ --src_category $src_cls$ \
--tgt_category $tgt_cls$ --use_transformer
Target-oracle. To train and finally evaluate the Target-oracle model with SL-module, simply run:
python train_SL_module.py --gpu_id $device_id$ --src_category $tgt_cls$ \
--tgt_category $tgt_cls$ --use_transformer
DA baselines. To train and finally evaluate two domain adaptation baselines (MMD and Deep CORAL), simply run:
python train_da_baselines.py --gpu_id $device_id$ --src_category $src_cls$ \
--tgt_category $tgt_cls$ --da_metric $mmd/coral$ \
--use_transformer
DL-VHD. To train and finally evaluate the DL-VHD model, simply run:
python train_dual_learner.py --gpu_id $device_id$ --src_category $src_cls$ \
--tgt_category $tgt_cls$ --use_transformer
Citation
If this work helps your research, you can kindly cite the following paper (will be updated when the ICCV paper is published).
@article{xu2021cross-category,
title={Cross-category Video Highlight Detection via Set-based Learning},
author={Xu, Minghao and Wang, Hang and Ni, Bingbing and Zhu, Riheng and Sun, Zhenbang and Wang, Changhu},
journal={arXiv preprint arXiv:2108.11770},
year={2021}
}
Could you tell me if there is something wrong with my training？
2 Oct 2, 2022
1 Jan 3, 2022
2 Dec 27, 2021
1 Jan 25, 2022
8.7k Dec 31, 2022
39 Nov 21, 2022
116.4k Jan 7, 2023
104 Dec 27, 2022
4 Jul 20, 2022
10 Jun 18, 2022
1 Oct 28, 2021
2 Oct 14, 2021
2 Jun 25, 2022
6 Dec 19, 2022
2 Aug 18, 2022
7 May 11, 2022
1 Oct 26, 2021
2 Jun 6, 2022