Official Implementation of SimIPU: Simple 2D Image and 3D Point Cloud Unsupervised Pre-Training for Spatial-Aware Visual Representations

Related tags

Deep Learning SimIPU
Overview

Official Implementation of SimIPU

  • SimIPU: Simple 2D Image and 3D Point Cloud Unsupervised Pre-Training for Spatial-Aware Visual Representations
  • Since the code is still waiting for release, if you have any question with reproduction, feel free to contact us. We will try our best to help you.
  • Currently, the core code of SimIPU is implemented in the commercial project. We are trying our best to make the code publicly available.
Comments
  • Question about augmentation

    Question about augmentation

    Hi, I'm a little confused about the data augmentation.

    1. How did you set img_aug when img_moco=True? It seems that we need an 'img_pipeline' in 'simipu_kitti.py', right?
    2. For 3D augmentation, it seems that it is done in this line. So the 3D augmentation is done based on the point features instead the raw points, right? If I want to try moco=True, how to set 3D augmentation? should I do this in the dataset building part? https://github.com/zhyever/SimIPU/blob/5b346e392c161a5e9fdde09b1692656bc7cd3faf/project_cl/decorator/inter_intro_decorator_moco_better.py#L394

    Looking forward to your reply. Many thanks.

    opened by sunnyHelen 2
  • error for env setup:ImportError: cannot import name 'ball_query_ext' from 'mmdet3d.ops.ball_query'

    error for env setup:ImportError: cannot import name 'ball_query_ext' from 'mmdet3d.ops.ball_query'

    Thanks for your insightful paper and clear code repo!

    Hi, I met with the ImportError: cannot import name 'ball_query_ext' from 'mmdet3d.ops.ball_query' when run the command bash tools/dist_train.sh project_cl/configs/simipu/simipu_kitti.py 1 --work_dir ./

    Do you know how to solve it?

    Traceback (most recent call last): File "tools/train.py", line 16, in from mmdet3d.apis import train_model File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/apis/init.py", line 1, in from .inference import (convert_SyncBN, inference_detector, File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/apis/inference.py", line 10, in from mmdet3d.core import (Box3DMode, DepthInstance3DBoxes, File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/core/init.py", line 2, in from .bbox import * # noqa: F401, F403 File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/core/bbox/init.py", line 4, in from .iou_calculators import (AxisAlignedBboxOverlaps3D, BboxOverlaps3D, File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/core/bbox/iou_calculators/init.py", line 1, in from .iou3d_calculator import (AxisAlignedBboxOverlaps3D, BboxOverlaps3D, File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/core/bbox/iou_calculators/iou3d_calculator.py", line 5, in from ..structures import get_box_type File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/core/bbox/structures/init.py", line 1, in from .base_box3d import BaseInstance3DBoxes File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/core/bbox/structures/base_box3d.py", line 5, in from mmdet3d.ops.iou3d import iou3d_cuda File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/ops/init.py", line 5, in from .ball_query import ball_query File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/ops/ball_query/init.py", line 1, in from .ball_query import ball_query File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/ops/ball_query/ball_query.py", line 4, in from . import ball_query_ext ImportError: cannot import name 'ball_query_ext' from 'mmdet3d.ops.ball_query' (/mnt/lustre/xxh/SimIPU-main/mmdet3d/ops/ball_query/init.py)

    I noticed that you once met with the same error. https://github.com/open-mmlab/mmdetection3d/issues/503#issuecomment-847618114

    So, I would like to ask for your help~ Hopefully you have a good solution. :)

    opened by JerryX1110 2
  • A question about eq5 and eq6

    A question about eq5 and eq6

    Thanks for your inspiring work. I have some wonder about eq5 and eq6. As far as I know, After eq5, f should be a tensor which is a global feature with shape (batchsize * 2048 * 1 * 1), how can you sample corresponding image features by projection location? After all, there's no spatial information in f anymore. Or maybe you got features from a previous layer of ResNet? Looking forward to your reply.

    opened by lianchengmingjue 2
  • A question about Tab.5 in Ablation Study

    A question about Tab.5 in Ablation Study

    Thanks for your excellent work first! I have a question about Tab.5 in Ablation Study. Why "Scratch" equals "SimIPU w/o inter-module ", which means that the intra-module is useless?

    opened by Trent-tangtao 1
  • Have you tried not to crop gradient of f^{\alpha} in eq7?

    Have you tried not to crop gradient of f^{\alpha} in eq7?

    Hi, I like your good work! I am wondering have you tried not to crop the gradient of $f^{\alpha}$ in eq7? If you crop the gradient, it seems like the pertaining of the point branch cannot learn anything from the image branch.

    opened by Hiusam 1
  • issues about create_data

    issues about create_data

    Hi, thanks for sharing your great work. I encounter some issues during creating data by running create_data.py First create reduced point cloud for training set [ ] 0/3712, elapsed: 0s, ETA:Traceback (most recent call last): File "tools/create_data.py", line 247, in
    out_dir=args.out_dir)
    File "tools/create_data.py", line 24, in kitti_data_prep
    kitti.create_reduced_point_cloud(root_path, info_prefix)
    File "/mnt/lustre/chenzhuo1/hzha/SimIPU/tools/data_converter/kitti_converter.py", line 374, in create_reduced_point_cloud
    _create_reduced_point_cloud(data_path, train_info_path, save_path)
    File "/mnt/lustre/chenzhuo1/hzha/SimIPU/tools/data_converter/kitti_converter.py", line 314, in _create_reduced_point_cloud
    count=-1).reshape([-1, num_features])
    ValueError: cannot reshape array of size 461536 into shape (6)

    It seems to set the num_features=4 and front_camera_id=2? in this line: https://github.com/zhyever/SimIPU/blob/5b346e392c161a5e9fdde09b1692656bc7cd3faf/tools/data_converter/kitti_converter.py#L291

    I assume doing this can solve the problem but encounter another problem when Create GT Database of KittiDataset
    [ ] 0/3712, elapsed: 0s, ETA:Traceback (most recent call last):
    File "tools/create_data.py", line 247, in
    out_dir=args.out_dir)
    File "tools/create_data.py", line 44, in kitti_data_prep
    with_bbox=True) # for moca
    File "/mnt/lustre/chenzhuo1/hzha/SimIPU/tools/data_converter/create_gt_database.py", line 275, in create_groundtruth_database
    P0 = np.array(example['P0']).reshape(4, 4)
    KeyError: 'P0'

    Can you help me figure out how to solve these issues?

    opened by sunnyHelen 21
Owner
Zhyever
Keep going.
Zhyever
Implementation of the "PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences" paper.

PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences Introduction Point cloud sequences are irregular and unordered in the spatial dimen

Hehe Fan 63 Dec 9, 2022
Implementation of the "Point 4D Transformer Networks for Spatio-Temporal Modeling in Point Cloud Videos" paper.

Point 4D Transformer Networks for Spatio-Temporal Modeling in Point Cloud Videos Introduction Point cloud videos exhibit irregularities and lack of or

Hehe Fan 101 Dec 29, 2022
Unofficial implementation of Point-Unet: A Context-Aware Point-Based Neural Network for Volumetric Segmentation

Point-Unet This is an unofficial implementation of the MICCAI 2021 paper Point-Unet: A Context-Aware Point-Based Neural Network for Volumetric Segment

Namt0d 9 Dec 7, 2022
pytorch implementation of "Contrastive Multiview Coding", "Momentum Contrast for Unsupervised Visual Representation Learning", and "Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination"

Unofficial implementation: MoCo: Momentum Contrast for Unsupervised Visual Representation Learning (Paper) InsDis: Unsupervised Feature Learning via N

Zhiqiang Shen 16 Nov 4, 2020
Official code release for "Learned Spatial Representations for Few-shot Talking-Head Synthesis" ICCV 2021

Official code release for "Learned Spatial Representations for Few-shot Talking-Head Synthesis" ICCV 2021

Moustafa Meshry 16 Oct 5, 2022
Revisiting Contrastive Methods for Unsupervised Learning of Visual Representations. [2021]

Revisiting Contrastive Methods for Unsupervised Learning of Visual Representations This repo contains the Pytorch implementation of our paper: Revisit

Wouter Van Gansbeke 80 Nov 20, 2022
Style-based Point Generator with Adversarial Rendering for Point Cloud Completion (CVPR 2021)

Style-based Point Generator with Adversarial Rendering for Point Cloud Completion (CVPR 2021) An efficient PyTorch library for Point Cloud Completion.

Microsoft 119 Jan 2, 2023
Synthetic LiDAR sequential point cloud dataset with point-wise annotations

SynLiDAR dataset: Learning From Synthetic LiDAR Sequential Point Cloud This is official repository of the SynLiDAR dataset. For technical details, ple

null 78 Dec 27, 2022
[ICCV 2021 Oral] SnowflakeNet: Point Cloud Completion by Snowflake Point Deconvolution with Skip-Transformer

This repository contains the source code for the paper SnowflakeNet: Point Cloud Completion by Snowflake Point Deconvolution with Skip-Transformer (ICCV 2021 Oral). The project page is here.

AllenXiang 65 Dec 26, 2022
Code for paper "ASAP-Net: Attention and Structure Aware Point Cloud Sequence Segmentation"

ASAP-Net This project implements ASAP-Net of paper ASAP-Net: Attention and Structure Aware Point Cloud Sequence Segmentation (BMVC2020). Overview We i

Hanwen Cao 26 Aug 25, 2022
Code of our paper "Contrastive Object-level Pre-training with Spatial Noise Curriculum Learning"

CCOP Code of our paper Contrastive Object-level Pre-training with Spatial Noise Curriculum Learning Requirement Install OpenSelfSup Install Detectron2

Chenhongyi Yang 21 Dec 13, 2022
ImageNet-CoG is a benchmark for concept generalization. It provides a full evaluation framework for pre-trained visual representations which measure how well they generalize to unseen concepts.

The ImageNet-CoG Benchmark Project Website Paper (arXiv) Code repository for the ImageNet-CoG Benchmark introduced in the paper "Concept Generalizatio

NAVER 23 Oct 9, 2022
3D AffordanceNet is a 3D point cloud benchmark consisting of 23k shapes from 23 semantic object categories, annotated with 56k affordance annotations and covering 18 visual affordance categories.

3D AffordanceNet This repository is the official experiment implementation of 3D AffordanceNet benchmark. 3D AffordanceNet is a 3D point cloud benchma

null 49 Dec 1, 2022
Part-Aware Data Augmentation for 3D Object Detection in Point Cloud

Part-Aware Data Augmentation for 3D Object Detection in Point Cloud This repository contains a reference implementation of our Part-Aware Data Augment

Jaeseok Choi 62 Jan 3, 2023
[ICCV 2021 Oral] PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformers

PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformers Created by Xumin Yu*, Yongming Rao*, Ziyi Wang, Zuyan Liu, Jiwen Lu, Jie Zhou

Xumin Yu 317 Dec 26, 2022
Maximum Spatial Perturbation for Image-to-Image Translation (Official Implementation)

MSPC for I2I This repository is by Yanwu Xu and contains the PyTorch source code to reproduce the experiments in our CVPR2022 paper Maximum Spatial Pe

null 51 Dec 14, 2022
Official Code for ICML 2021 paper "Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline"

Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline Ankit Goyal, Hei Law, Bowei Liu, Alejandro Newell, Jia Deng Internati

Princeton Vision & Learning Lab 115 Jan 4, 2023