Self-Supervised Pillar Motion Learning for Autonomous Driving (CVPR 2021)

Related tags

Deep Learning autonomous-driving lidar-point-cloud self-supervised-learning nuscenes point-cloud-motion
Overview

Self-Supervised Pillar Motion Learning for Autonomous Driving

Chenxu Luo, Xiaodong Yang, Alan Yuille
Self-Supervised Pillar Motion Learning for Autonomous Driving, CVPR 2021
[Paper] [Poster] [YouTube]

Getting Started

Installation

Install PyTorch, PyTorch3D, Apex, nuScenes Devkit

Data Preparation

python tools/create_data nuscenes_data_prep --root_path /path/to/nuscenes

Our optical flow model used for the cross-sensor regularization is available here.

Training

python -m torch.distributed.launch --nproc_per_node=8 ./tools/train.py configs/nusc_pillarmotion.py --work_dir experiments/pillarmotion/

Citation

Please cite the following paper if this repo helps your research:

@InProceedings{Luo_2021_CVPR,
    author    = {Luo, Chenxu and Yang, Xiaodong and Yuille, Alan},
    title     = {Self-Supervised Pillar Motion Learning for Autonomous Driving},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2021},
    pages     = {3183-3192}
}

License

Copyright (C) 2021 QCraft. All rights reserved. Licensed under the CC BY-NC-SA 4.0 (Attribution-NonCommercial-ShareAlike 4.0 International). The code is released for academic research use only. For commercial use, please contact [email protected].

Comments
  • optf[cam_id == cid, :2] = flo[cam_coord[cam_id == cid, 1].astype(np.int32), points[cam_id == cid, 0].astype(np.int32)] IndexError: index 1147174912 is out of bounds for axis 0 with size 900

    optf[cam_id == cid, :2] = flo[cam_coord[cam_id == cid, 1].astype(np.int32), points[cam_id == cid, 0].astype(np.int32)] IndexError: index 1147174912 is out of bounds for axis 0 with size 900

    I installed the PWC-DC network according to https://github.com/NVlabs/PWC-Net. When I ran “python tools/create_data nuscenes_data_prep --root_path /path/to/nuscenes” command, I encountered the following problems.

    Traceback (most recent call last): File "motion/datasets/nuscenes/flow.py", line 654, in res = save_flow(info, root_path) File "motion/datasets/nuscenes/flow.py", line 638, in save_flow optf[cam_id == cid, :2] = flo[cam_coord[cam_id == cid, 1].astype(np.int32), points[cam_id == cid, 0].astype(np.int32)] IndexError: index 1147174912 is out of bounds for axis 0 with size 900

    ========================================== The following are some variables I checked, which are really out of range. I hope you can help me. Thank you very much. # print(points.shape) # (34720, 5) # print(optf.shape) # (34720, 3) print(cam_coord) """ [[-1082130432 -1082130432] [-1082130432 -1082130432] [-1082130432 -1082130432] ... [ 1150836736 1133019136] [ 1150844928 1131479040] [ 1150861312 1129316352]] """ # print(cam_id.shape) # (34720,) # print(flo.shape) # (900, 1600, 2) # print(cam_coord.shape) # (34720, 2) # print(points.shape) # (34720, 5) # print(cam_coord[cam_id == cid, 1]) # [1147174912 1147158528 1146519552 ... 1129381888 1146650624 1146044416] # print(cam_coord[cam_id == cid, 1].astype(np.int32).shape) # (2516,) # print(cam_coord[cam_id == cid, 1].astype(np.int32)) # [1147174912 1147158528 1146519552... 1129381888 1146650624 1146044416]

    opened by liangchunlan 2
  • Question about motion/datasets/pipelines/loading/read_sweep

    Question about motion/datasets/pipelines/loading/read_sweep

    Hi!, thanks for your good project to the lidar motion. But I have a question about the motion/datasets/pipelines/loading/read_sweep. There is a parameter "transform" in read_sweep, but when call read_sweept in motion/datasets/pipelines/loading/ line 70, the parameter is missed. How can I fix this missing? Is the parameter "transform" in read_sweep for factorizing ego-motion out?

    opened by JXZxiao5 0
  • nuscenes dataset loading error

    nuscenes dataset loading error "assert hasattr(self.dataset, "flag") AssertionError"

    First, prepared the data using "data/v1.0-mini" for test. python tools/create_data nuscenes_data_prep --root_path /path/to/nuscenes. Got files: train_anno = "data/nuscenes/results/infos_train_05sweeps_twoframe_withcam.pkl" val_anno = "data/nuscenes/results/infos_val_05sweeps_twoframe_withcam.pkl" test_anno = "data/nuscenes/results/infos_test_05sweeps_twoframe_withcam.pkl" But, when I run train.py, following error occured. Traceback (most recent call last): File "./tools/train.py", line 133, in <module> main() File "./tools/train.py", line 122, in main train_detector( File "/nfs/private/01_Code/pillar-motion/motion/torchie/apis/train.py", line 258, in train_detector data_loaders = [ File "/nfs/private/01_Code/pillar-motion/motion/torchie/apis/train.py", line 259, in <listcomp> build_dataloader( File "/nfs/private/01_Code/pillar-motion/motion/datasets/loader/build_loader.py", line 34, in build_dataloader sampler = DistributedGroupSampler(dataset, batch_size, world_size, rank) File "/nfs/private/01_Code/pillar-motion/motion/datasets/loader/sampler.py", line 166, in __init__ assert hasattr(self.dataset, "flag") AssertionError

    How can I solve this? THANKS!

    opened by taoyanzhuce001 6
  • there are a lot of bugs! Basically can not get normal results

    there are a lot of bugs! Basically can not get normal results

    There are a lot of bugs! Basically can not get normal results. If you want to improve the credibility and reputation of the paper, it is recommended to open source code that can reproduce the results of the article

    opened by VisonSpace 0
  • Question about the PWCnet

    Question about the PWCnet

    hi~, chenxu, this is a good job. But when I reproduce this project, i found some mistake in this repo, and I have slove some. But there is also one question I need your help. I test the PWCNet you provided, and I found that the flow may not right. For instance, left image and right image are as fellows, and I use the flow produced bu PWCNet to warp right image to left. The results is bad as show.

    image leat and image right: image

    warped image right: image

    opened by tyjiang1997 0
  • Question about the data augmention

    Question about the data augmention

    Hi!, thanks for your good project to the lidar motion. But I have a question about the data augmention. I notice that the "input_points" are augmented with global flip and scaling, but the "source_points" and "target_points" do not do this augmetation. Maybe the predicted motion can not transform "source_points" to "target_points" to compute precise chamfer loss. Is this a bug?

    opened by tyjiang1997 1
  • No train_cfg and test_cfg

    No train_cfg and test_cfg

    opened by zxczrx123 2
Owner
QCraft
QCraft
GitHub
Graph Self-Attention Network for Learning Spatial-Temporal Interaction Representation in Autonomous Driving

GSAN Introduction Code for paper GSAN: Graph Self-Attention Network for Learning Spatial-Temporal Interaction Representation in Autonomous Driving, wh

YE Luyao 6 Oct 27, 2022
[CVPR'21] Multi-Modal Fusion Transformer for End-to-End Autonomous Driving

TransFuser This repository contains the code for the CVPR 2021 paper Multi-Modal Fusion Transformer for End-to-End Autonomous Driving. If you find our

null 695 Jan 5, 2023
[CVPR 21] Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.

Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting, CVPR 2021. Ayan Kumar Bhunia, Pinaki nath Chowdhury, Yongxin Yan

Ayan Kumar Bhunia 44 Dec 12, 2022
This solves the autonomous driving issue which is supported by deep learning technology. Given a video, it splits into images and predicts the angle of turning for each frame.

Self Driving Car An autonomous car (also known as a driverless car, self-driving car, and robotic car) is a vehicle that is capable of sensing its env

Sagor Saha 4 Sep 4, 2021
[CVPR 2021] "The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models" Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, Zhangyang Wang

The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models Codes for this paper The Lottery Tickets Hypo

VITA 59 Dec 28, 2022
Repository to run object detection on a model trained on an autonomous driving dataset.

Autonomous Driving Object Detection on the Raspberry Pi 4 Description of Repository This repository contains code and instructions to configure the ne

Ethan 51 Nov 17, 2022
Official Repo for Ground-aware Monocular 3D Object Detection for Autonomous Driving

Visual 3D Detection Package: This repo aims to provide flexible and reproducible visual 3D detection on KITTI dataset. We expect scripts starting from

Yuxuan Liu 305 Dec 19, 2022
RTS3D: Real-time Stereo 3D Detection from 4D Feature-Consistency Embedding Space for Autonomous Driving

RTS3D: Real-time Stereo 3D Detection from 4D Feature-Consistency Embedding Space for Autonomous Driving (AAAI2021). RTS3D is efficiency and accuracy s

null 71 Nov 29, 2022
Uncertainty-aware Semantic Segmentation of LiDAR Point Clouds for Autonomous Driving

SalsaNext: Fast, Uncertainty-aware Semantic Segmentation of LiDAR Point Clouds for Autonomous Driving Abstract In this paper, we introduce SalsaNext f

null 308 Jan 4, 2023
One Million Scenes for Autonomous Driving

ONCE Benchmark This is a reproduced benchmark for 3D object detection on the ONCE (One Million Scenes) dataset. The code is mainly based on OpenPCDet.

null 148 Dec 28, 2022
[ICCV'21] NEAT: Neural Attention Fields for End-to-End Autonomous Driving

NEAT: Neural Attention Fields for End-to-End Autonomous Driving Paper | Supplementary | Video | Poster | Blog This repository is for the ICCV 2021 pap

null 254 Jan 2, 2023
Code repository for Semantic Terrain Classification for Off-Road Autonomous Driving

BEVNet Datasets Datasets should be put inside data/. For example, data/semantic_kitti_4class_100x100. Training BEVNet-S Example: cd experiments bash t

(Brian) JoonHo Lee 24 Dec 12, 2022
An unofficial personal implementation of UM-Adapt, specifically to tackle joint estimation of panoptic segmentation and depth prediction for autonomous driving datasets.

Semisupervised Multitask Learning This repository is an unofficial and slightly modified implementation of UM-Adapt[1] using PyTorch. This code primar

Abhinav Atrishi 11 Nov 25, 2022
Unofficial PyTorch implementation of "RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Driving" (ECCV 2020)

RTM3D-PyTorch The PyTorch Implementation of the paper: RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Driving (ECCV 2020

Nguyen Mau Dzung 271 Nov 29, 2022
The Self-Supervised Learner can be used to train a classifier with fewer labeled examples needed using self-supervised learning.

Published by SpaceML • About SpaceML • Quick Colab Example Self-Supervised Learner The Self-Supervised Learner can be used to train a classifier with

SpaceML 92 Nov 30, 2022
(CVPR 2022) A minimalistic mapless end-to-end stack for joint perception, prediction, planning and control for self driving.

LAV Learning from All Vehicles Dian Chen, Philipp Krähenbühl CVPR 2022 (also arXiV 2203.11934) This repo contains code for paper Learning from all veh

Dian Chen 300 Dec 15, 2022
Dense Contrastive Learning (DenseCL) for self-supervised representation learning, CVPR 2021.

Dense Contrastive Learning for Self-Supervised Visual Pre-Training This project hosts the code for implementing the DenseCL algorithm for se

Xinlong Wang 491 Jan 3, 2023
Kaggle Lyft Motion Prediction for Autonomous Vehicles 4th place solution

Lyft Motion Prediction for Autonomous Vehicles Code for the 4th place solution of Lyft Motion Prediction for Autonomous Vehicles on Kaggle. Discussion

null 44 Jun 27, 2022
Video Autoencoder: self-supervised disentanglement of 3D structure and motion

Video Autoencoder: self-supervised disentanglement of 3D structure and motion This repository contains the code (in PyTorch) for the model introduced

null 157 Dec 22, 2022