Applying CLIP to Point Cloud Recognition.

Related tags

Deep Learning PointCLIP
Overview

PointCLIP: Point Cloud Understanding by CLIP

This repository is an official implementation of the paper 'PointCLIP: Point Cloud Understanding by CLIP'.

Introduction

PointCLIP is the first to apply CLIP for point cloud recognition, which transfers 2D pre-trained knowledge into 3D domains. Specifically, we encode a point cloud by projecting it into multi-view depth maps without rendering, and aggregate the view-wise predictions for zero-shot classification.

On top of that, we design an inter-view adapter to further enhance the few-shot performance, and explore the effectiveness of muti-knowledge ensembling.

Implementation

Coming soon!

Comments
  • The zero-shot classification acc is lower than origin paper

    The zero-shot classification acc is lower than origin paper

    Thanks for your wonderful work which is helpful for my research. I found that when I use 'bash zeroshot.sh', the classification acc is about 19.33% (20.18% in your paper). I have tried many times. But the gap still exists. Please help me, thank you again !

    opened by wangyePHD 1
  • Zero Learning Question

    Zero Learning Question

    Hello, Zero Learning is to make predictions on categories you don't see. CLIP self supervised training on 400 million picture text pairs will inevitably see categories you don't see. Is this also called Zero Learning? Thank you for your patience.

    opened by DuanhaoranCC 1
  • How to ensemble pointclip and other point model?

    How to ensemble pointclip and other point model?

    Hello author.I am very interested in your research.Could you tell me how to ensemble pointclip and other point model such as pointnet? Could you give me some tips?I'm looking forward to you reply.

    opened by suyzhi 1
  • the question about the scanobjectnn dataset

    the question about the scanobjectnn dataset

    Hi, thanks for the interesting work! I noticed that, for the zeroshot classification, only the modelnet40 dataset part of code was released, is there any chance the scanobjectnn part of code will be released thus can be reproduced? Thanks

    opened by Tycho-Xue 0
  • 点云的feature与clip中text feature的问题

    点云的feature与clip中text feature的问题

    想了解一下有考虑过用pointnet之类的学到的point cloud global feature加上全连接层做finetune然后与clip中的text feature比较这样的尝试吗，还是说因为clip中image encoder和text encoder学到的特征是对齐的，所以直接考虑了2d depth maps projection的思路，如果有考虑过前者的话，是效果不好吗？

    opened by jiatianzhi 1
  • Apply for the pretrained weights on other few-shot learning

    Apply for the pretrained weights on other few-shot learning

    Thanks for your wonderful work, it is very interesting and helpful to my research. Therefore, I want to apply the other few-shot learning pre-trained weights not only 16-shot as follows: image

    opened by wangye8899 1
Owner
Renrui Zhang
Pre-PhD candidate at MMLab, CUHK.
Renrui Zhang
GitHub
Implementation of the "PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences" paper.

PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences Introduction Point cloud sequences are irregular and unordered in the spatial dimen

Hehe Fan 63 Dec 9, 2022
Implementation of the "Point 4D Transformer Networks for Spatio-Temporal Modeling in Point Cloud Videos" paper.

Point 4D Transformer Networks for Spatio-Temporal Modeling in Point Cloud Videos Introduction Point cloud videos exhibit irregularities and lack of or

Hehe Fan 101 Dec 29, 2022
Synthetic LiDAR sequential point cloud dataset with point-wise annotations

SynLiDAR dataset: Learning From Synthetic LiDAR Sequential Point Cloud This is official repository of the SynLiDAR dataset. For technical details, ple

null 78 Dec 27, 2022
[ICCV 2021 Oral] SnowflakeNet: Point Cloud Completion by Snowflake Point Deconvolution with Skip-Transformer

This repository contains the source code for the paper SnowflakeNet: Point Cloud Completion by Snowflake Point Deconvolution with Skip-Transformer (ICCV 2021 Oral). The project page is here.

AllenXiang 65 Dec 26, 2022
FuseDream: Training-Free Text-to-Image Generationwith Improved CLIP+GAN Space OptimizationFuseDream: Training-Free Text-to-Image Generationwith Improved CLIP+GAN Space Optimization

FuseDream This repo contains code for our paper (paper link): FuseDream: Training-Free Text-to-Image Generation with Improved CLIP+GAN Space Optimizat

XCL 191 Dec 31, 2022
CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIP

CLIP-GEN [简体中文][English] 本项目在萤火二号集群上用 PyTorch 实现了论文 《CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIP》。 CLIP-GEN 是一个 Language-F

null 75 Dec 29, 2022
A list of papers about point cloud based place recognition, also known as loop closure detection in SLAM (processing)

A list of papers about point cloud based place recognition, also known as loop closure detection in SLAM (processing)

Xin Kong 17 May 16, 2021
PointNetVLAD: Deep Point Cloud Based Retrieval for Large-Scale Place Recognition, CVPR 2018

PointNetVLAD: Deep Point Cloud Based Retrieval for Large-Scale Place Recognition PointNetVLAD: Deep Point Cloud Based Retrieval for Large-Scale Place

Mikaela Uy 294 Dec 12, 2022
Official pytorch implementation of "DSPoint: Dual-scale Point Cloud Recognition with High-frequency Fusion"

DSPoint Official implementation of "DSPoint: Dual-scale Point Cloud Recognition with High-frequency Fusion". Paper link: https://arxiv.org/abs/2111.10

Ziyao Zeng 10 Nov 24, 2021
Repo for "Benchmarking Robustness of 3D Point Cloud Recognition against Common Corruptions" https://arxiv.org/abs/2201.12296

Benchmarking Robustness of 3D Point Cloud Recognition against Common Corruptions This repo contains the dataset and code for the paper Benchmarking Ro

Jiachen Sun 168 Dec 29, 2022
Image morphing without reference points by applying warp maps and optimizing over them.

Differentiable Morphing Image morphing without reference points by applying warp maps and optimizing over them. Differentiable Morphing is machine lea

Alex K 380 Dec 19, 2022
Face2webtoon - Despite its importance, there are few previous works applying I2I translation to webtoon.

Despite its importance, there are few previous works applying I2I translation to webtoon. I collected dataset from naver webtoon 연애혁명 and tried to transfer human faces to webtoon domain.

이상윤 64 Oct 19, 2022
SparseML is a libraries for applying sparsification recipes to neural networks with a few lines of code, enabling faster and smaller models

SparseML is a toolkit that includes APIs, CLIs, scripts and libraries that apply state-of-the-art sparsification algorithms such as pruning and quantization to any neural network. General, recipe-driven approaches built around these algorithms enable the simplification of creating faster and smaller models for the ML performance community at large.

Neural Magic 1.5k Dec 30, 2022
Fuzzification helps developers protect the released, binary-only software from attackers who are capable of applying state-of-the-art fuzzing techniques

About Fuzzification Fuzzification helps developers protect the released, binary-only software from attackers who are capable of applying state-of-the-

gts3.org (SSLab@Gatech) 55 Oct 25, 2022
Code for "PV-RAFT: Point-Voxel Correlation Fields for Scene Flow Estimation of Point Clouds", CVPR 2021

PV-RAFT This repository contains the PyTorch implementation for paper "PV-RAFT: Point-Voxel Correlation Fields for Scene Flow Estimation of Point Clou

Yi Wei 43 Dec 5, 2022
Unofficial implementation of Point-Unet: A Context-Aware Point-Based Neural Network for Volumetric Segmentation

Point-Unet This is an unofficial implementation of the MICCAI 2021 paper Point-Unet: A Context-Aware Point-Based Neural Network for Volumetric Segment

Namt0d 9 Dec 7, 2022
Point-NeRF: Point-based Neural Radiance Fields

Point-NeRF: Point-based Neural Radiance Fields Project Sites | Paper | Primary c

Qiangeng Xu 662 Jan 1, 2023
Not All Points Are Equal: Learning Highly Efficient Point-based Detectors for 3D LiDAR Point Clouds (CVPR 2022, Oral)

Not All Points Are Equal: Learning Highly Efficient Point-based Detectors for 3D LiDAR Point Clouds (CVPR 2022, Oral) This is the official implementat

Yifan Zhang 259 Dec 25, 2022
Pytorch implementation of PCT: Point Cloud Transformer

PCT: Point Cloud Transformer This is a Pytorch implementation of PCT: Point Cloud Transformer.

Yi_Zhang 265 Dec 22, 2022