This is the official pytorch implementation for our ICCV 2021 paper "TRAR: Routing the Attention Spans in Transformers for Visual Question Answering" on VQA Task

Related tags

Deep Learning ERASOR
Overview

🌈 ERASOR (RA-L'21 with ICRA Option)

Official page of "ERASOR: Egocentric Ratio of Pseudo Occupancy-based Dynamic Object Removal for Static 3D Point Cloud Map Building", which is accepted by RA-L with ICRA'21 option [Demo Video].

overview

We provide all contents including

  • Source code of ERASOR
  • All outputs of the State-of-the-arts
  • Visualization
  • Calculation code of Preservation Rate/Rejection Rate

So enjoy our codes! :)

Contact: Hyungtae Lim ([email protected])

Advisor: Hyun Myung ([email protected])

Contents

  1. Test Env.
  2. Requirements
  3. How to Run ERASOR
  4. Calculate PR/RR
  5. Benchmark
  6. Run Your Own Code
  7. Visualization of All the State-of-the-arts
  8. Citation

Test Env.

The code is tested successfully at

  • Linux 18.04 LTS
  • ROS Melodic

Requirements

ROS Setting

  • Install ROS on a machine.
  • Also, jsk-visualization is required to visualize Scan Ratio Test (SRT) status.
sudo apt-get install ros-melodic-jsk-recognition
sudo apt-get install ros-melodic-jsk-common-msgs
sudo apt-get install ros-melodic-jsk-rviz-plugins

Buildg Our Package

mkdir -p ~/catkin_ws/src
cd ~/catkin_ws/src
git clone https://github.com/LimHyungTae/ERASOR.Official.git
cd .. && catkin build erasor 

Python Setting

  • Our metric calculation for PR/RR code is implemented by python2.7
  • To run the python code, following pakages are necessary: pypcd, tqdm, scikit-learn, and tabulate
pip install pypcd
pip install tqdm	
pip install scikit-learn
pip install tabulate

Prepared dataset

  • Download the preprocessed KITTI data encoded into rosbag.
  • The downloading process might take five minutes or so. All rosbags requires total 2.3G of storage space
wget https://urserver.kaist.ac.kr/publicdata/erasor/rosbag/00_4390_to_4530_w_interval_2_node.bag
wget https://urserver.kaist.ac.kr/publicdata/erasor/rosbag/01_150_to_250_w_interval_1_node.bag
wget https://urserver.kaist.ac.kr/publicdata/erasor/rosbag/02_860_to_950_w_interval_2_node.bag
wget https://urserver.kaist.ac.kr/publicdata/erasor/rosbag/05_2350_to_2670_w_interval_2_node.bag
wget https://urserver.kaist.ac.kr/publicdata/erasor/rosbag/07_630_to_820_w_interval_2_node.bag

Description of Preprocessed Rosbag Files

  • Please note that the rosbag consists of node. Refer to msg/node.msg.
  • Note that each label of the point is assigned in intensity for the sake of convenience.
  • And we set the following classes are dynamic classes:
# 252: "moving-car"
# 253: "moving-bicyclist"
# 254: "moving-person"
# 255: "moving-motorcyclist"
# 256: "moving-on-rails"
# 257: "moving-bus"
# 258: "moving-truck"
# 259: "moving-other-vehicle"
  • Please refer to std::vector DYNAMIC_CLASSES in our code :).

How to Run ERASOR

We will explain how to run our code on seq 05 of the KITTI dataset as an example.

Step 1. Build naive map

kittimapgen

  • Set the following parameters in launch/mapgen.launch.
    • target_rosbag: The name of target rosbag, e.g. 05_2350_to_2670_w_interval_2_node.bag
    • save_path: The path where the naively accumulated map is saved.
  • Launch mapgen.launch and play corresponding rosbag on the other bash as follows:
roscore # (Optional)
roslaunch erasor mapgen.launch
rosbag play 05_2350_to_2670_w_interval_2_node.bag
  • Then, dense map and voxelized map are auto-saved at the save path. Note that the dense map is used to fill corresponding labels (HERE). The voxelized map will be an input of step 2 as a naively accumulated map.

Step 2. Run ERASOR erasor

  • Set the following parameters in config/seq_05.yaml.

    • initial_map_path: The path of naively accumulated map
    • save_path: The path where the filtered static map is saved.
  • Run the following command for each bash.

roscore # (Optional)
roslaunch erasor run_erasor.launch target_seq:="05"
rosbag play 05_2350_to_2672_w_interval_2_node.bag
  • IMPORTANT: After finishing running ERASOR, run the following command to save the static map as a pcd file on another bash.
  • "0.2" denotes voxelization size.
rostopic pub /saveflag std_msgs/Float32 "data: 0.2"
  • Then, you can see the printed command as follows:

fig_command

  • The results will be saved under the save_path folder, i.e. $save_path$/05_result.pcd.

Calculate PR/RR

You can check our results directly.

  • First, download all pcd materials.
wget https://urserver.kaist.ac.kr/publicdata/erasor/erasor_paper_pcds.zip
unzip erasor_paper_pcds.zip

Then, run the analysis code as follows:

python analysis.py --gt $GT_PCD_PATH$ --est $EST_PCD_PATH$

E.g,

python analysis.py --gt /home/shapelim/erasor_paper_pcds/gt/05_voxel_0_2.pcd --est /home/shapelim/erasor_paper_pcds/estimate/05_ERASOR.pcd

NOTE: For estimating PR/RR, more dense pcd file, which is generated in the mapgen.launch procedure, is better to estimate PR/RR precisely.

Benchmark

  • Error metrics are a little bit different from those in the paper:

    Seq. PR [%] RR [%]
    00 91.72 97.00
    01 91.93 94.63
    02 81.08 99.11
    05 86.98 97.88
    07 92.00 98.33
  • But we provide all pcd files! Don't worry. See Visualization of All the State-of-the-arts Section.

Run Your Own Code

⚠️ TBU: The code is already in this repository, yet the explanation is incomplete.

Visualization of All the State-of-the-arts

  • First, download all pcd materials.
wget https://urserver.kaist.ac.kr/publicdata/erasor/erasor_paper_pcds.zip
unzip erasor_paper_pcds.zip
  • Set parameters in config/viz_params.yaml correctly

    • abs_dir: The absolute directory of pcd directory
    • seq: Target sequence (00, 01, 02, 05, or 07)
  • After setting the parameters, launch following command:

roslaunch erasor compare_results.launch

Citation

If you use our code or method in your work, please consider citing the following:

@article{lim2021erasor,
title={ERASOR: Egocentric Ratio of Pseudo Occupancy-Based Dynamic Object Removal for Static 3D Point Cloud Map Building},
author={Lim, Hyungtae and Hwang, Sungwon and Myung, Hyun},
journal={IEEE Robotics and Automation Letters},
volume={6},
number={2},
pages={2272--2279},
year={2021},
publisher={IEEE}
}
Comments
  • Assertion `cloud.points.size () == cloud.width * cloud.height' failed.

    Assertion `cloud.points.size () == cloud.width * cloud.height' failed.

    Hi @LimHyungTae ! Thank you for your excellent work. I ran the demo on my computer and the result looks great. However, when I tried ERASOR on my own dataset, I encountered the following error:

    main_in_your_env: /usr/include/pcl-1.8/pcl/conversions.h:252: void pcl::toPCLPointCloud2(const pcl::PointCloud<PointT>&, pcl::PCLPointCloud2&) [with PointT = pcl::PointXYZI]: Assertion `cloud.points.size () == cloud.width * cloud.height' failed.
    

    This happens on a specific frame. I checked the corresponding .pcd file but it looked alright. I also tried this suggestion but it did not work. Could you please give me some hints on how to solve this, thanks!

    opened by zxl19 8
  • catkin build error: can not find erasor/node.h.

    catkin build error: can not find erasor/node.h.

    Thanks for your great work! According to your introduction, I encountered the following problem. I can't find node.h from the source file. @LimHyungTae image Could you please give me some help? 🥺

    opened by zxczrx123 2
  • ERROR, when i run

    ERROR, when i run "roslaunch erasor run_erasor_in_your_env.launch"

    main_in_your_env: /usr/include/eigen3/Eigen/src/Core/util/XprHelper.h:110: Eigen::internal::variable_if_dynamic<T, Value>::variable_if_dynamic(T) [with T = long int; int Value = 3]: Assertionv == T(Value)' failed. [main_in_your_env-2] process has died [pid 32096, exit code -6, cmd /home/dt/ROS_WS/erasor_ws/devel/lib/erasor/main_in_your_env __name:=main_in_your_env __log:=/home/dt/.ros/log/5305fe14-348b-11ec-9ebf-081f7134fb94/main_in_your_env-2.log]. log file: /home/dt/.ros/log/5305fe14-348b-11ec-9ebf-081f7134fb94/main_in_your_env-2*.log`

    opened by llxClover 2
  • how to run this on on single part in the whole map

    how to run this on on single part in the whole map

    hi , i recently tried your work and merged it to our work and it was excellent! thanks for your wonderful contribution to the dynamic objects removing! and i wonder if there is a way to just run your ERASOR to remove objects in just a part of a whole map bag, which means i don't have to run it on my whole map since it may inevitably delete some static pointcloud. looking forward to your reply!

    opened by yuhang9803 2
  • can not find compare_results.launch

    can not find compare_results.launch

    Thank you for sharing your interesting work, @LimHyungTae But I can't find this command:

    roslaunch erasor compare_results.launch
    

    Can you update this missng file?

    opened by crankler 2
  • License info missing

    License info missing

    The package.xml of this package has the default TODO entry for the license used, which makes it quite difficult use it in a legally consistent manner. Would be great if a valid license entry could be added.

    opened by skohlbr 1
  • use HDL 32E Velodyne's  data

    use HDL 32E Velodyne's data

    Hi @LimHyungTae, thank you for your great work! I have one question. If i use Velodyne 32E's data, what parameters should be changed? The parameters of Lidar are below. image

    opened by pytorch1100 1
  • how can i get the dynamic points and static points by the results!

    how can i get the dynamic points and static points by the results!

    first, i get the whole map by mapgen.launch .second , i run the run_erasor.launch to get the result map by erasor. then, how can i get the dynamic points cloud, and the result map by erasor should the static point cloud.

    opened by xiaowei520-lwl 0
  • Fix two potentional problems, check commit for details

    Fix two potentional problems, check commit for details

    This pull request fix two problems here, if you tried in your own dataset, maybe you will also get this error pop out.

    1. if the ground point is zero, origin repo's code will stop kill by itself and print this error:

      I20221202 20:08:49.231433 79705 erasor.cpp:619] ground0
      main_in_your_env_ros: /usr/include/pcl-1.10/pcl/conversions.h:251: void pcl::toPCLPointCloud2(const pcl::PointCloud<PointT>&, pcl::PCLPointCloud2&) [with PointT = pcl::PointXYZI]: Assertion `cloud.points.size () == cloud.width * cloud.height' failed.
      [main_in_your_env_ros-2] process has died [pid 79705, exit code -6, cmd /home/kin/workspace/ERASOR_ws/devel/lib/erasor/main_in_your_env_ros __name:=main_in_your_env_ros __log:=/home/kin/.ros/log/a4aa26be-7274-11ed-b5b3-ef4eaaa7a243/main_in_your_env_ros-2.log].
      log file: /home/kin/.ros/log/a4aa26be-7274-11ed-b5b3-ef4eaaa7a243/main_in_your_env_ros-2*.log
      
    2. poses read the matrix construction will cause potential problem, if you run on your dataset:

      tf4x4_cam.topLeftCorner<3, 3>(0, 0) = q.toRotationMatrix();
      tf4x4_cam.topRightCorner(3, 1) = ts.vector();
      

      Error like this one

      /usr/include/eigen3/Eigen/src/Core/util/XprHelper.h:113: Eigen::internal::variable_if_dynamic<T, Value>::variable_if_dynamic(T) [with T = long int; int Value = 3]: Assertion `v == T(Value)' failed.
      
    opened by Kin-Zhang 1
  • Remove much more static points(false positive) than expectation

    Remove much more static points(false positive) than expectation

    Hello, I'm sorry to bother you. I test your method in my own datasets, but I found the false positive points is very large. In the raw map, the stat. pts is 844762, and in the estimated map, the est.stat.pts is 475379, which leading the preservation rate is very low. Is it because of the parameter setting that causes this situation? How to deal with it? I'm waiting for your reply! Best!

    opened by fentuoli 0
  • Calculate PR/RR

    Calculate PR/RR

    Dear Author, I'm sorry to bother you, I have some questions about the calculation of pr/rr. I notice the gt pcd files in the erasor_paper_pcds are generated by your mapgen.cpp method, so when you generate the estimated maps by other methods such as Removert, did you take any steps such as ensuring the consistency of downsample_voxel_size to get more accurate results? I'm waiting for your reply! Best Wishes!

    opened by fentuoli 1
  • Error when run 'roslaunch erasor run_erasor_in_your_env_vel16.launch'

    Error when run 'roslaunch erasor run_erasor_in_your_env_vel16.launch'

    [ INFO] [1668568723.312462663]: PASS! [ INFO] [1668568723.411885393]: 1303th frame is comming [ INFO] [1668568723.490956884]: 1014759=1014759| 138259 + 876500 [ INFO] [1668568723.607537384]: Extracting VoI takes 0.1828s main_in_your_env_ros: /usr/include/pcl-1.8/pcl/conversions.h:252: void pcl::toPCLPointCloud2(const pcl::PointCloud&, pcl::PCLPointCloud2&) [with PointT = pcl::PointXYZI]: Assertion `cloud.points.size () == cloud.width * cloud.height' failed.

    opened by MIke-1118 0
  • How to build a whole map

    How to build a whole map

    Hi ! This is such a nice work and thank you very much for your contribution ! My question is that how can I run the whole KITTI05 instead of a part of the dataset? I would be very grateful if you can give me some advice! Thank you for seeing my question!

    opened by MIke-1118 2
  • Eigen::internal::variable_if_dynamic<T, Value>::variable_if_dynamic(T) [with T = long int; int Value = 3]: Assertion `v == T(Value)' failed.

    Eigen::internal::variable_if_dynamic::variable_if_dynamic(T) [with T = long int; int Value = 3]: Assertion `v == T(Value)' failed.

    main_in_your_env_ros: /usr/include/eigen3/Eigen/src/Core/util/XprHelper.h:110: Eigen::internal::variable_if_dynamic<T, Value>::variable_if_dynamic(T) [with T = long int; int Value = 3]: Assertion `v == T(Value)' failed. [main_in_your_env_ros-1] process has died [pid 32655, exit code -6, cmd /home/dsj/catkin_erasor/devel/lib/erasor/main_in_your_env_ros __name:=main_in_your_env_ros __log:=/home/dsj/.ros/log/4f1bf28e-2dac-11ed-a109-d05099fb0c81/main_in_your_env_ros-1.log]. log file: /home/dsj/.ros/log/4f1bf28e-2dac-11ed-a109-d05099fb0c81/main_in_your_env_ros-1*.log

    opened by xiaoxiaolu1111 5
Owner
Hyungtae Lim
Ph.D Candidate of URL lab. @ KAIST, South Korea
Hyungtae Lim
Official PyTorch implementation for Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers, a novel method to visualize any Transformer-based network. Including examples for DETR, VQA.

PyTorch Implementation of Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers 1 Using Colab Please notic

Hila Chefer 489 Jan 7, 2023
An efficient PyTorch implementation of the winning entry of the 2017 VQA Challenge.

Bottom-Up and Top-Down Attention for Visual Question Answering An efficient PyTorch implementation of the winning entry of the 2017 VQA Challenge. The

Hengyuan Hu 731 Jan 3, 2023
Pytorch implementation of winner from VQA Chllange Workshop in CVPR'17

2017 VQA Challenge Winner (CVPR'17 Workshop) pytorch implementation of Tips and Tricks for Visual Question Answering: Learnings from the 2017 Challeng

Mark Dong 166 Dec 11, 2022
PyTorch implementation of our ICCV 2021 paper, Interpretation of Emergent Communication in Heterogeneous Collaborative Embodied Agents.

PyTorch implementation of our ICCV 2021 paper, Interpretation of Emergent Communication in Heterogeneous Collaborative Embodied Agents.

Saim Wani 4 May 8, 2022
[CVPR 2021] Counterfactual VQA: A Cause-Effect Look at Language Bias

Counterfactual VQA (CF-VQA) This repository is the Pytorch implementation of our paper "Counterfactual VQA: A Cause-Effect Look at Language Bias" in C

Yulei Niu 94 Dec 3, 2022
TAP: Text-Aware Pre-training for Text-VQA and Text-Caption, CVPR 2021 (Oral)

TAP: Text-Aware Pre-training TAP: Text-Aware Pre-training for Text-VQA and Text-Caption by Zhengyuan Yang, Yijuan Lu, Jianfeng Wang, Xi Yin, Dinei Flo

Microsoft 61 Nov 14, 2022
Neural Module Network for VQA in Pytorch

Neural Module Network (NMN) for VQA in Pytorch Note: This is NOT an official repository for Neural Module Networks. NMN is a network that is assembled

Harsh Trivedi 111 Nov 24, 2022
Implementation for our ICCV 2021 paper: Dual-Camera Super-Resolution with Aligned Attention Modules

DCSR: Dual Camera Super-Resolution Implementation for our ICCV 2021 oral paper: Dual-Camera Super-Resolution with Aligned Attention Modules paper | pr

Tengfei Wang 110 Dec 20, 2022
Implementation for our ICCV 2021 paper: Dual-Camera Super-Resolution with Aligned Attention Modules

DCSR: Dual Camera Super-Resolution Implementation for our ICCV 2021 oral paper: Dual-Camera Super-Resolution with Aligned Attention Modules paper | pr

Tengfei Wang 110 Dec 20, 2022
official Pytorch implementation of ICCV 2021 paper FuseFormer: Fusing Fine-Grained Information in Transformers for Video Inpainting.

FuseFormer: Fusing Fine-Grained Information in Transformers for Video Inpainting By Rui Liu, Hanming Deng, Yangyi Huang, Xiaoyu Shi, Lewei Lu, Wenxiu

null 77 Dec 27, 2022
Official PyTorch Implementation of paper "Deep 3D Mask Volume for View Synthesis of Dynamic Scenes", ICCV 2021.

Deep 3D Mask Volume for View Synthesis of Dynamic Scenes Official PyTorch Implementation of paper "Deep 3D Mask Volume for View Synthesis of Dynamic S

Ken Lin 17 Oct 12, 2022
Official Pytorch implementation of the paper "Action-Conditioned 3D Human Motion Synthesis with Transformer VAE", ICCV 2021

ACTOR Official Pytorch implementation of the paper "Action-Conditioned 3D Human Motion Synthesis with Transformer VAE", ICCV 2021. Please visit our we

Mathis Petrovich 248 Dec 23, 2022
Official code for our ICCV paper: "From Continuity to Editability: Inverting GANs with Consecutive Images"

GANInversion_with_ConsecutiveImgs Official code for our ICCV paper: "From Continuity to Editability: Inverting GANs with Consecutive Images" https://a

QingyangXu 38 Dec 7, 2022
PyTorch implementation of our Adam-NSCL algorithm from our CVPR2021 (oral) paper "Training Networks in Null Space for Continual Learning"

Adam-NSCL This is a PyTorch implementation of Adam-NSCL algorithm for continual learning from our CVPR2021 (oral) paper: Title: Training Networks in N

Shipeng Wang 34 Dec 21, 2022
This is an official PyTorch implementation of Task-Adaptive Neural Network Search with Meta-Contrastive Learning (NeurIPS 2021, Spotlight).

NeurIPS 2021 (Spotlight): Task-Adaptive Neural Network Search with Meta-Contrastive Learning This is an official PyTorch implementation of Task-Adapti

Wonyong Jeong 15 Nov 21, 2022
Official implementation of the paper Vision Transformer with Progressive Sampling, ICCV 2021.

Vision Transformer with Progressive Sampling This is the official implementation of the paper Vision Transformer with Progressive Sampling, ICCV 2021.

yuexy 123 Jan 1, 2023
Official implementation of the ICCV 2021 paper "Conditional DETR for Fast Training Convergence".

The DETR approach applies the transformer encoder and decoder architecture to object detection and achieves promising performance. In this paper, we handle the critical issue, slow training convergence, and present a conditional cross-attention mechanism for fast DETR training. Our approach is motivated by that the cross-attention in DETR relies highly on the content embeddings and that the spatial embeddings make minor contributions, increasing the need for high-quality content embeddings and thus increasing the training difficulty.

null 281 Dec 30, 2022
The Official Implementation of the ICCV-2021 Paper: Semantically Coherent Out-of-Distribution Detection.

SCOOD-UDG (ICCV 2021) This repository is the official implementation of the paper: Semantically Coherent Out-of-Distribution Detection Jingkang Yang,

Jake YANG 62 Nov 21, 2022
Official implementation of the ICCV 2021 paper: "The Power of Points for Modeling Humans in Clothing".

The Power of Points for Modeling Humans in Clothing (ICCV 2021) This repository contains the official PyTorch implementation of the ICCV 2021 paper: T

Qianli Ma 158 Nov 24, 2022