Starter code for the ICCV 2021 paper, 'Detecting Invisible People'

Overview

Detecting Invisible People

[ICCV 2021 Paper] [Website]

Tarasha Khurana, Achal Dave, Deva Ramanan

Introduction

This repository contains code for Detecting Invisible People. We extend the original DeepSORT algorithm to localize people even while they are completely occluded in a video. See the arXiv preprint for more information.

Dependencies

Create a conda environment with the given environment.yml file.

conda env create -f environment.yml

Preprocessing

The code expects the directory structure of your dataset in the MOT Challenge data format, which is approximately like the following:

MOT17/
-- train/
---- seq_01/
------ img1/
------ img1Depth/
------ gt/
------ det/
...
-- test/
---- seq_02/
------ img1/
------ img1Depth/
------ det/

The folder img1Depth stores the normalized disparity in .npy format. See Note. Originally, the paper runs the method on depth given by the MegaDepth depth estimator.

Given the above folder structure, generate the appearance features for your detections as described in the DeepSORT repository.

Running the method

The script run_forecast_filtering.sh will run the method with hyperparameters used in the paper. It will produce output .txt files in the MOT Challenge submission format. The bashscript has support for computing the metrics, but this has not been verified. Run the bashscript like the following:

bash run_forecast_filtering.sh experimentName

Note that in order to speed up code release, the dataset, preprocessed detections and output file paths are hardcoded in the files and will have to be manually changed.

Citing Detecting Invisible People

If you find this code useful in your research, please consider citing the following paper:

@inproceedings{khurana2021detecting,
  title={{Detecting Invisible People}},
  author={Khurana, Tarasha and Dave, Achal and Ramanan, Deva},
  booktitle={{IEEE/CVF International Conference on Computer Vision (ICCV)}},
  year={2021}
}

Warning

This is only the starter code that has not been cleaned for release. It currently only has verified support for running the method described in Detecting Invisible People, with the output tracks written in the MOT Challenge submission format. Although Top-k metric's code has been provided, this codebase does not guarantee support for the metric yet.

The hope is that you are able to benchmark this method for your CVPR 2022 submission and compute your own metrics on the method's output. If the method code does not work, please open an issue.

Note

Although it is easy to run any monocular depth estimator and store their output (usually given as disparity) in an .npy file, I have added a script in tools/demo_images.py which can save the .npy files for you. Note that this script should be run after setting up the MegaDepth codebase and copying this file to its root directory. I will likely also release my own depth maps for the MOT17 dataset over the Halloween weekend.

If you try to run the metrics, I have given my groundtruth JSON (as expected by pycocotools).

You might also like...
Code release for ICCV 2021 paper
Code release for ICCV 2021 paper "Anticipative Video Transformer"

Anticipative Video Transformer Ranked first in the Action Anticipation task of the CVPR 2021 EPIC-Kitchens Challenge! (entry: AVT-FB-UT) [project page

Official code release for ICCV 2021 paper SNARF: Differentiable Forward Skinning for Animating Non-rigid Neural Implicit Shapes.
Official code release for ICCV 2021 paper SNARF: Differentiable Forward Skinning for Animating Non-rigid Neural Implicit Shapes.

Official code release for ICCV 2021 paper SNARF: Differentiable Forward Skinning for Animating Non-rigid Neural Implicit Shapes.

Demo code for ICCV 2021 paper
Demo code for ICCV 2021 paper "Sensor-Guided Optical Flow"

Sensor-Guided Optical Flow Demo code for "Sensor-Guided Optical Flow", ICCV 2021 This code is provided to replicate results with flow hints obtained f

Code for ICCV 2021 paper
Code for ICCV 2021 paper "HuMoR: 3D Human Motion Model for Robust Pose Estimation"

Code for ICCV 2021 paper "HuMoR: 3D Human Motion Model for Robust Pose Estimation"

Code for the ICCV 2021 Workshop paper: A Unified Efficient Pyramid Transformer for Semantic Segmentation.

Unified-EPT Code for the ICCV 2021 Workshop paper: A Unified Efficient Pyramid Transformer for Semantic Segmentation. Installation Linux, CUDA=10.0,

Code for
Code for "Layered Neural Rendering for Retiming People in Video."

Layered Neural Rendering in PyTorch This repository contains training code for the examples in the SIGGRAPH Asia 2020 paper "Layered Neural Rendering

Official implementation of the paper Vision Transformer with Progressive Sampling, ICCV 2021.
Official implementation of the paper Vision Transformer with Progressive Sampling, ICCV 2021.

Vision Transformer with Progressive Sampling This is the official implementation of the paper Vision Transformer with Progressive Sampling, ICCV 2021.

PyTorch implementation of paper: AdaAttN: Revisit Attention Mechanism in Arbitrary Neural Style Transfer, ICCV 2021.
PyTorch implementation of paper: AdaAttN: Revisit Attention Mechanism in Arbitrary Neural Style Transfer, ICCV 2021.

AdaAttN: Revisit Attention Mechanism in Arbitrary Neural Style Transfer [Paper] [PyTorch Implementation] [Paddle Implementation] Overview This reposit

Homepage of paper: Paint Transformer: Feed Forward Neural Painting with Stroke Prediction, ICCV 2021.
Homepage of paper: Paint Transformer: Feed Forward Neural Painting with Stroke Prediction, ICCV 2021.

Paint Transformer: Feed Forward Neural Painting with Stroke Prediction [Paper] [PaddlePaddle Implementation] Homepage of paper: Paint Transformer: Fee

Comments
  • Some questions about the project

    Some questions about the project

    I apologize for my previous rudeness, but I do have some issues that I would like to discuss rationally with you. as follows. First, I think the code in the paper that is open sourced for others to study should at least be available or error-free, not just use github as our own code repository.

    There are a lot of code writing errors in this project. 1.Where is the definition or introduction of the function get_mask_from_bbox() in line 83 of the tracker.py file in the deep_sort folder? 2.What’s the meaning of the parameter “tn=self.tn” of the function update() in line 95? 3.Where is the definition or call of the function compute_mean_depth_from_mask() in line 434? 4.Where is the definition of the variable only_extrapolate in line 161 of the tracker_mask.py file? 5.Does the function project() require a parameter called temporal_noise in line 301 of the kalman_filter.py file? 6.and so on Please double check and update the code carefully when you have time. Also, I have a question, are the results stated in the paper based on this project??

    Second, the project is not coding standard. 1.There are a lot of unused imports in the project, such as “import ast” in detection.py, cv2 in kalman_filter.py, and so on. 2.In addition, in the kalman_filter.py file, the type of tn was originally int, later in the line 92 tn became a dict? Shouldn't the correct way to write line 80 be “if not tn”?

    Finally,I previous said that the necessary comments are missing in the code, like 1.what exactly is the meaning of tn mentioned above? Whether it is a glyph or a dict with specific meaning is not given in the code. 2.In line 132 of Kalman_filter.py, the function project() requires 5 parameters. Why are there no corresponding annotations for the other 3 parameters except the mean and covariance parameters that were originally in the deep_sort project? There are many such cases, please check carefully when you have time, and add this part of the note if you can.

    Detecting Invisible People is a funny stuff, I wanted to peruse the code for this project, but it was kind of bad, and reading it made me so irritable, hence i raise yesterday's issue. I think not only we must learn to be a rational person, also we should be rational and rigorous with our own research and works. The above are some of the questions I have raised. I am looking forward to your reply.

    opened by 158627746 1
  • S**t code!

    S**t code!

    I think this is something that only the author can understand, or not.Why is there no manual?and there are no comments in the your code, Reading this makes me cranky!!!!!!

    opened by 158627746 1
  • Visualization code

    Visualization code

    Thanks for your great work, is there any code available to visualize the tracks in 3D as shown in the website? https://www.cs.cmu.edu/~tkhurana/imgs/teaser.gif

    opened by brjathu 0
Owner
Tarasha Khurana
Tarasha Khurana
Implementation for the paper SMPLicit: Topology-aware Generative Model for Clothed People (CVPR 2021)

SMPLicit: Topology-aware Generative Model for Clothed People [Project] [arXiv] License Software Copyright License for non-commercial scientific resear

Enric Corona 225 Dec 13, 2022
Code for ICCV2021 paper SPEC: Seeing People in the Wild with an Estimated Camera

SPEC: Seeing People in the Wild with an Estimated Camera [ICCV 2021] SPEC: Seeing People in the Wild with an Estimated Camera, Muhammed Kocabas, Chun-

Muhammed Kocabas 187 Dec 26, 2022
Code repository for the paper "Tracking People with 3D Representations"

Tracking People with 3D Representations Code repository for the paper "Tracking People with 3D Representations" (paper link) (project site). Jathushan

Jathushan Rajasegaran 77 Dec 3, 2022
Starter kit for getting started in the Music Demixing Challenge.

Music Demixing Challenge - Starter Kit ?? Challenge page This repository is the Music Demixing Challenge Submission template and Starter kit! Clone th

AIcrowd 106 Dec 20, 2022
Code for ICCV 2021 paper "Distilling Holistic Knowledge with Graph Neural Networks"

HKD Code for ICCV 2021 paper "Distilling Holistic Knowledge with Graph Neural Networks" cifia-100 result The implementation of compared methods are ba

Wang Yucheng 30 Dec 18, 2022
code for ICCV 2021 paper 'Generalized Source-free Domain Adaptation'

G-SFDA Code (based on pytorch 1.3) for our ICCV 2021 paper 'Generalized Source-free Domain Adaptation'. [project] [paper]. Dataset preparing Download

Shiqi Yang 84 Dec 26, 2022
Code for ICCV 2021 paper: ARAPReg: An As-Rigid-As Possible Regularization Loss for Learning Deformable Shape Generators..

ARAPReg Code for ICCV 2021 paper: ARAPReg: An As-Rigid-As Possible Regularization Loss for Learning Deformable Shape Generators.. Installation The cod

Bo Sun 132 Nov 28, 2022
Code for the ICCV 2021 paper "Pixel Difference Networks for Efficient Edge Detection" (Oral).

Pixel Difference Convolution This repository contains the PyTorch implementation for "Pixel Difference Networks for Efficient Edge Detection" by Zhuo

Alex 236 Dec 21, 2022
Sync2Gen Code for ICCV 2021 paper: Scene Synthesis via Uncertainty-Driven Attribute Synchronization

Sync2Gen Code for ICCV 2021 paper: Scene Synthesis via Uncertainty-Driven Attribute Synchronization 0. Environment Environment: python 3.6 and cuda 10

Haitao Yang 62 Dec 30, 2022
Code for the paper "Spatio-temporal Self-Supervised Representation Learning for 3D Point Clouds" (ICCV 2021)

Spatio-temporal Self-Supervised Representation Learning for 3D Point Clouds This is the official code implementation for the paper "Spatio-temporal Se

Hesper 63 Jan 5, 2023