3D AffordanceNet is a 3D point cloud benchmark consisting of 23k shapes from 23 semantic object categories, annotated with 56k affordance annotations and covering 18 visual affordance categories.

Related tags

Deep Learning AffordanceNet
Overview

3D AffordanceNet

This repository is the official experiment implementation of 3D AffordanceNet benchmark.

3D AffordanceNet is a 3D point cloud benchmark consisting of 23k shapes from 23 semantic object categories, annotated with 56k affordance annotations and covering 18 visual affordance categories.

This repository implements two baseline methods: PointNet++ and DGCNN on four proposed affordance understanding tasks: Full-Shape, Partial-View, Rotation-Invariant, Semi-Supervised Affordance Estimation.

You can reproduce the performances described in the origin paper by simply running a command down below.

[CVPR 2021 Paper] [Dataset Download Link] [Project Page]

GroundTruth

Requirements

All the codes are tested in the following environment:

  • Linux (tested on Ubuntu 16.04)
  • Python 3.7+
  • PyTorch 1.0.1
  • Gorilla-Core
  • CUDA 10.0 or higher

You can install the required packages by running the following command:

pip install -r requirement.txt

To install the cuda kernel, go to models/pointnet2_ops and run the following command:

python setup.py build_ext --inplace

Quick Start

The following set up is for DGCNN, you can change to PointNet++ accordingly.

First download the whole dataset from here and extract the files to the data_root, then modify the dataset data_root in configuration(full-shape for example), the dataset data_root should obey the data structure below:

data_root
    ├── task_train_data.pkl
    ├── task_val_data.pkl
    └── task_test_data.pkl

Then to train a model from scratch:

python train.py config/dgcnn/estimation_cfg.py --work_dir TPATH_TO_LOG_DIR --gpu 0,1

After training, to test a model:

python test.py config/dgcnn/estimation_cfg.py --work_dir PATH_TO_LOG_DIR --gpu 0,1 --checkpoint PATH_TO_CHECKPOINT

Currently Support

  • Models
    • DGCNN
    • PointNet++
  • Tasks
    • Full-Shape Affordance Estimation
    • Partial-View Affordance Estimation
    • Rotation-Invariant Affordance Estimation
    • Semi-Supervised Affordance Estimation
Comments
  • error: command 'g++' failed with exit status 1

    error: command 'g++' failed with exit status 1

    Hi, When I execute "python setup.py build_ext --inplace" there is one error like follow picture, Is it because the g++ version is different? 微信图片_20210414154121

    opened by mxmdpc 2
  • Can you provide any pretrained models?

    Can you provide any pretrained models?

    I trained and testes with the default full-shape configuration of dgcnn and pointnet++, and could not reproduce the performances described in the original paper.

    I trained all 200 epochs and tested with the last and best checkpoint. For dgcnn I get:

    grasp_AP = 0.2556759690225669
contain_AP = 0.2571292042795441
lift_AP = 0.023838588310033604
openable_AP = 0.1360015272654039
layable_AP = 0.12459241370490073
sittable_AP = 0.27206526362544564
support_AP = 0.16712899651185892
wrap_grasp_AP = 0.08832214710297487
pourable_AP = 0.34237704497247096
move_AP = 0.19473959311947647
displaY_AP = 0.16057008441655551
pushable_AP = 0.03182707396969131
pull_AP = 0.013906940482448393
listen_AP = 0.10797679116717496
wear_AP = 0.09977950447691103
press_AP = 0.03501589515410332
cut_AP = 0.4247068143872306
stab_AP = 0.015207772739159924
Test :: test mAP: 0.152826, test mAUC: 0.570680, test maIOU: 0.008561, test MSE: 0.094147

    For pointnet++ I get:

    grasp_AP = 0.2850018957704849
contain_AP = 0.3970870684151038
lift_AP = 0.2605200101147898
openable_AP = 0.14212084464199967
layable_AP = 0.20583821697005833
sittable_AP = 0.7337801898061207
support_AP = 0.46620184133153686
wrap_grasp_AP = 0.1210342362727236
pourable_AP = 0.30525177684139226
move_AP = 0.29397890070768157
displaY_AP = 0.36314677703229237
pushable_AP = 0.04884707447625411
pull_AP = 0.03213688947134047
listen_AP = 0.09369795453349795
wear_AP = 0.07568989410259043
press_AP = 0.1392326344367762
cut_AP = 0.34916580702635086
stab_AP = 0.5896400841975753
Test :: test mAP: 0.272354, test mAUC: 0.728240, test maIOU: 0.056233, test MSE: 2.891442

    Can you provide some pretrained models?

    opened by zzilch 2
  • Is it safe to change the default epoch when resume from checkpoint?

    Is it safe to change the default epoch when resume from checkpoint?

    With the current code, the start epoch is set to 0 by default when I resume from a checkpoint. Let's say my checkpoint model is the one at epoch 40 and I want to resume it. And I want the logger to show that my start epoch is 40, just for convenience. Would it be safe, does it have any bad effects on the optimizer or the learning strategy?

    opened by toannguyen1904 1
  • Just Say Thank U

    Just Say Thank U

    Thank U for your amazing work. My research direction is afforedance but is 2D affordance segmentation at present. But I always have a question is how to label the data? Can your give some question to me?

    Thank you. Regards

    opened by mxmdpc 1
  • Dice Loss

    Dice Loss

    I find inconsistencies between the formula of the loss function in the paper and its implementation in your code. Besides that, I see your definition of Dice Loss differs from the Dice Loss definition in the original paper that is based on Dice Coefficient as follows image Could you give me some explanation?

    opened by toannguyen1904 0
  • Questions related to the dataset annotation and multi-gpu training results

    Questions related to the dataset annotation and multi-gpu training results

    Hi, thanks for the great work! After running some experiments, I found two issues.

    1. Some of the annotations are not correct. Here I show some examples (left model: prediction, right model: GT). image (The top two shelves have no annotation of "contain") image (The pourable annotation lies on the bottom of the bottle) image (The grasp annotation lies on the bottleneck) image image (Grasp annotations are quite different for visually similar bottles)

    2. The results vary a lot when using different numbers of GPU, and it seems single GPU got the best performance. What's the reason?

      | PointNet++ -- | -- 8 gpu | Test :: test mAP: 0.433250, test mAUC: 0.856608, test maIOU: 0.057118, test MSE: 0.071557 4 gpu | Test :: test mAP: 0.437279, test mAUC: 0.845381, test maIOU: 0.116516, test MSE: 0.065208 1 gpu | Test :: test mAP: 0.481856, test mAUC: 0.876145, test maIOU: 0.183885, test MSE: 0.058885

    opened by Reagan1311 3
Owner
Research lab focusing on CV, ML, and AI
null
GitHub
Point Cloud Denoising input segmentation output raw point-cloud valid/clear fog rain de-noised Abstract Lidar sensors are frequently used in environme

Point Cloud Denoising input segmentation output raw point-cloud valid/clear fog rain de-noised Abstract Lidar sensors are frequently used in environme

null 75 Nov 24, 2022
Tensorflow AffordanceNet and AffContext implementations

AffordanceNet and AffContext This is tensorflow AffordanceNet and AffContext implementations. Both are implemented and tested with tensorflow 2.3. The

Beatriz Pérez 6 Dec 1, 2022
O2O-Afford: Annotation-Free Large-Scale Object-Object Affordance Learning (CoRL 2021)

O2O-Afford: Annotation-Free Large-Scale Object-Object Affordance Learning Object-object Interaction Affordance Learning. For a given object-object int

Kaichun Mo 26 Nov 4, 2022
Cerberus Transformer: Joint Semantic, Affordance and Attribute Parsing

Cerberus Transformer: Joint Semantic, Affordance and Attribute Parsing Paper Introduction Multi-task indoor scene understanding is widely considered a

null 62 Dec 5, 2022
🐦 Opytimizer is a Python library consisting of meta-heuristic optimization techniques.

Opytimizer: A Nature-Inspired Python Optimizer Welcome to Opytimizer. Did you ever reach a bottleneck in your computational experiments? Are you tired

Gustavo Rosa 546 Dec 31, 2022
MVP Benchmark for Multi-View Partial Point Cloud Completion and Registration

MVP Benchmark: Multi-View Partial Point Clouds for Completion and Registration [NEWS] 2021-07-12 [NEW ?? ] The submission on Codalab starts! 2021-07-1

PL 93 Dec 21, 2022
Style-based Point Generator with Adversarial Rendering for Point Cloud Completion (CVPR 2021)

Style-based Point Generator with Adversarial Rendering for Point Cloud Completion (CVPR 2021) An efficient PyTorch library for Point Cloud Completion.

Microsoft 119 Jan 2, 2023
Implementation of the "PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences" paper.

PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences Introduction Point cloud sequences are irregular and unordered in the spatial dimen

Hehe Fan 63 Dec 9, 2022
Implementation of the "Point 4D Transformer Networks for Spatio-Temporal Modeling in Point Cloud Videos" paper.

Point 4D Transformer Networks for Spatio-Temporal Modeling in Point Cloud Videos Introduction Point cloud videos exhibit irregularities and lack of or

Hehe Fan 101 Dec 29, 2022
[ICCV 2021 Oral] SnowflakeNet: Point Cloud Completion by Snowflake Point Deconvolution with Skip-Transformer

This repository contains the source code for the paper SnowflakeNet: Point Cloud Completion by Snowflake Point Deconvolution with Skip-Transformer (ICCV 2021 Oral). The project page is here.

AllenXiang 65 Dec 26, 2022
Behind the Curtain: Learning Occluded Shapes for 3D Object Detection

Behind the Curtain: Learning Occluded Shapes for 3D Object Detection Acknowledgement We implement our model, BtcDet, based on [OpenPcdet 0.3.0]. Insta

Qiangeng Xu 163 Dec 19, 2022
Implementation based on Paper - Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

Implementation based on Paper - Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

HamasKhan 3 Jul 8, 2022
Co-mining: Self-Supervised Learning for Sparsely Annotated Object Detection, AAAI 2021.

Co-mining: Self-Supervised Learning for Sparsely Annotated Object Detection This repository is an official implementation of the AAAI 2021 paper Co-mi

MEGVII Research 20 Dec 7, 2022
Semantic Segmentation for Real Point Cloud Scenes via Bilateral Augmentation and Adaptive Fusion (CVPR 2021)

Semantic Segmentation for Real Point Cloud Scenes via Bilateral Augmentation and Adaptive Fusion (CVPR 2021) This repository is for BAAF-Net introduce

null 90 Dec 29, 2022
[CVPR 2021] Few-shot 3D Point Cloud Semantic Segmentation

Few-shot 3D Point Cloud Semantic Segmentation Created by Na Zhao from National University of Singapore Introduction This repository contains the PyTor

null 117 Dec 27, 2022
Perturbed Self-Distillation: Weakly Supervised Large-Scale Point Cloud Semantic Segmentation (ICCV2021)

Perturbed Self-Distillation: Weakly Supervised Large-Scale Point Cloud Semantic Segmentation (ICCV2021) This is the implementation of PSD (ICCV 2021),

null 12 Dec 12, 2022
PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud, CVPR 2019.

PointRCNN PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud Code release for the paper PointRCNN:3D Object Proposal Generation a

Shaoshuai Shi 1.5k Dec 27, 2022
Investigating Attention Mechanism in 3D Point Cloud Object Detection (arXiv 2021)

Investigating Attention Mechanism in 3D Point Cloud Object Detection (arXiv 2021) This repository is for the following paper: "Investigating Attention

null 52 Nov 19, 2022
Part-Aware Data Augmentation for 3D Object Detection in Point Cloud

Part-Aware Data Augmentation for 3D Object Detection in Point Cloud This repository contains a reference implementation of our Part-Aware Data Augment

Jaeseok Choi 62 Jan 3, 2023