Code for "CloudAAE: Learning 6D Object Pose Regression with On-line Data Synthesis on Point Clouds" @ICRA2021

Last update: Nov 14, 2022

Related tags

Overview

CloudAAE

This is an tensorflow implementation of "CloudAAE: Learning 6D Object Pose Regression with On-line Data Synthesis on Point Clouds"

Files

log: directory to store log files during training.
losses: loss functions for training.
models: a python file defining model structure.
object_model_tfrecord: full object models for data synthesizing and visualization purpose.
tf_ops: tensorflow implementation of sampling operations (credit: Haoqiang Fan, Charles R. Qi).
trained_network: a trained network.
utils: utility files for defining model structure.
ycb_video_data_tfRecords: synthetic training data and real test data for the YCB video dataset.
evaluate_cloudAAE_ycbv.py: script for testing object 6d pose estimation with a trained network on test set in YCB video dataset.
train_cloudAAE_ycbv.py: script for training a network on synthetic data for YCB objects.

Requirements

Tensorflow-GPU (tested with 1.12.0)
transforms3d
open3d for visualization

Test a trained network

Testing data in tfrecord format is available

Download zip file
Unzip and place all files in ycb_video_data_tfRecords/test_real/

After activate tensorflow

python evaluate_cloudAAE_ycbv.py --trained_model trained_network/20200908-204328/model.ckpt --batch_size 1 --target_cls 0

--trained_model: directory to trained model (*.ckpt).
--batch_size: 1.
--target_class: target class for pose estimation.
Translation prediction is in unit meter.
Rotation prediction is in axis-angle format.

Result

If you turn on visualization with b_visual=True, you will see the following displays which are partially observed point cloud segments (red) overlaid with object model (green) with pose estimates. The reconstructed point cloud is also presented (blue).
The coordinate is the object coordinate, object segment is viewed in the camera coordinate

Train a network

Training data is created synthetically using 3D object model and 6D poses.

The 6D pose and class id of target object are in ycb_video_data_tfRecords/train_syn/
The data synthesis pipeline takes the target 3D object model and creates a segment of the object in the desired 6D pose. Below is two examples of synthetic segment (red), two real segments (red) are also shown for comparison.

Run script

python train_cloudAAE_ycbv.py

Log files and trained model is store in log

Citation

If you use this code in an academic context, please consider cite the paper:

BiBTeX:

@inproceedings{gao2020cloudpose,
      title={CloudAAE: Learning 6D Object Pose Regression with On-line Data
Synthesis on Point Clouds},
      author={G. Gao, M. Lauri, X. Hu, J. Zhang and S. Frintrop},
      booktitle={ICRA},
      year={2021}
    }

Link to Paper

TBA

Acknowledgement

The building block for this system is PointNet and Dynamic Graph.

Comments

How to generate synthetic training data

Hi Ge,

thanks for sharing this great work! I would like to test it with my own objects. But I couldn't find the codes for generating the training data. For ycb dataset, you have already saved them in tf_records datatype in ycb_video_data_tfRecords/train_syn/. Could you share more details about this part?

Best, Jianxiang

opened by JianxiangFENG 4
linemod dataset

hi,when i run linemod_to_tfrecord_small.py,i encounter a problem--tensorflow.python.framework.errors_impl.InvalidArgumentError: 0-th value returned by pyfunc_0 is int32, but expects int64 it seems to the tensorflow function:tf.py_func,rasie such wrong. i think it is probably the following tf.int64 rasie the wrong. x['frame_id'] = tf.py_func(read_frame_id, [x['rgb']], tf.int64) i have no idea to fix it,hope your reply.

opened by tonglily 2
Test on retrained model have errors !

Hi，Ge I have trained a model follow by your Train a network script : python train_cloudAAE_ycbv.py . and I have success training a model. but when i test the model use evaluate_cloudAAE_ycbv.py script , have a errors happen :

NFO - 2021-08-15 15:36:03,823 - tf_logging - Restoring parameters from trained_network/20210731-231714/model.ckpt 2021-08-15 15:36:03.885536: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key dgcnn1/bn/6d_pose/dgcnn1/bn/moments/Squeeze/ExponentialMovingAverage not found in checkpoint Traceback (most recent call last): File "/root/anaconda3/envs/caae/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call return fn(*args) File "/root/anaconda3/envs/caae/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/root/anaconda3/envs/caae/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.NotFoundError: Key dgcnn1/bn/6d_pose/dgcnn1/bn/moments/Squeeze/ExponentialMovingAverage not found in checkpoint [[{{node save/RestoreV2}} = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]] [[{{node save/RestoreV2/_53}} = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_75_save/RestoreV2", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]]

what does this cause? How to fixed the problem . and wish your reply . thanks !

opened by welen-zhou 2
Can this project run on RTX3090?

Dear GeeeG： First of all, thank you for your contribution to this project, which has brought great inspiration to my research work. Recently I tried to run this project on RTX3090, but often encountered unexpected errors, I learned that 30-series graphics cards do not support the CUDA11.x version below, I found in another issue you are using CUDA11.3 to run this project, I would like to ask you how to do it. Thank you again for your contribution to this project.

opened by Xenos24R 2
object_model_tfrecord

hi,when I want to run this work using my own dataset,but I don't know how to make the file of object_model_tfrecord. can you tell me how to do this file? hope your rely!

opened by tonglily 1
Hoe to get experiment results on LM and LMO datatset?

Dear Ge: Thanks for sharing so nice work and releasing your code.
However, I cannot find the train and test code to get experiment results on LM and LMO datatset. Would you mind giving some guidelines to reproduce experiment results in terms of these dataset?

opened by ZJU-PLP 1

Owner

Gee

I like point cloud.

GitHub

This is the official source code for SLATE. We provide the code for the model, the training code, and a dataset loader for the 3D Shapes dataset. This code is implemented in Pytorch.

SLATE This is the official source code for SLATE. We provide the code for the model, the training code and a dataset loader for the 3D Shapes dataset.

66 Dec 26, 2022

TensorFlow code for the neural network presented in the paper: "Structural Language Models of Code" (ICML'2020)

SLM: Structural Language Models of Code This is an official implementation of the model described in: "Structural Language Models of Code" [PDF] To ap

73 Nov 6, 2022

Inference code for "StylePeople: A Generative Model of Fullbody Human Avatars" paper. This code is for the part of the paper describing video-based avatars.

NeuralTextures This is repository with inference code for paper "StylePeople: A Generative Model of Fullbody Human Avatars" (CVPR21). This code is for

Visual Understanding Lab @ Samsung AI Center Moscow

18 Oct 6, 2022

A code generator from ONNX to PyTorch code

onnx-pytorch Generating pytorch code from ONNX. Currently support onnx==1.9.0 and torch==1.8.1. Installation From PyPI pip install onnx-pytorch From

94 Jan 6, 2023

This is the code for our KILT leaderboard submission to the T-REx and zsRE tasks. It includes code for training a DPR model then continuing training with RAG.

KGI (Knowledge Graph Induction) for slot filling This is the code for our KILT leaderboard submission to the T-REx and zsRE tasks. It includes code fo

72 Jan 6, 2023

Convert Python 3 code to CUDA code.

Py2CUDA Convert python code to CUDA. Usage To convert a python file say named py_file.py to CUDA, run python generate_cuda.py --file py_file.py --arch

3 Jul 14, 2021

Empirical Study of Transformers for Source Code & A Simple Approach for Handling Out-of-Vocabulary Identifiers in Deep Learning for Source Code

Transformers for variable misuse, function naming and code completion tasks The official PyTorch implementation of: Empirical Study of Transformers fo

56 Nov 15, 2022

Reference implementation of code generation projects from Facebook AI Research. General toolkit to apply machine learning to code, from dataset creation to model training and evaluation. Comes with pretrained models.

This repository is a toolkit to do machine learning for programming languages. It implements tokenization, dataset preprocessing, model training and m

408 Jan 1, 2023

Code for the prototype tool in our paper "CoProtector: Protect Open-Source Code against Unauthorized Training Usage with Data Poisoning".

CoProtector Code for the prototype tool in our paper "CoProtector: Protect Open-Source Code against Unauthorized Training Usage with Data Poisoning".

1 Oct 26, 2021

Low-code/No-code approach for deep learning inference on devices

EzEdgeAI A concept project that uses a low-code/no-code approach to implement deep learning inference on devices. It provides a componentized framewor

7 Apr 5, 2022

Code for all the Advent of Code'21 challenges mostly written in python

Advent of Code 21 Code for all the Advent of Code'21 challenges mostly written in python. They are not necessarily the best or fastest solutions but j

4 May 26, 2022

Code to use Augmented Shapiro Wilks Stopping, as well as code for the paper "Statistically Signifigant Stopping of Neural Network Training"

This codebase is being actively maintained, please create and issue if you have issues using it Basics All data files are included under losses and ea

32 Nov 9, 2021

Opinionated code formatter, just like Python's black code formatter but for Beancount

beancount-black Opinionated code formatter, just like Python's black code formatter but for Beancount Try it out online here Features MIT licensed - b

16 Oct 11, 2022

a delightful machine learning tool that allows you to train, test and use models without writing code

igel A delightful machine learning tool that allows you to train/fit, test and use models without writing code Note I'm also working on a GUI desktop

3k Jan 5, 2023

Pytorch Lightning code guideline for conferences

Deep learning project seed Use this seed to start new deep learning / ML projects. Built in setup.py Built in requirements Examples with MNIST Badges

1k Jan 2, 2023

Automatically Build Multiple ML Models with a Single Line of Code. Created by Ram Seshadri. Collaborators Welcome. Permission Granted upon Request.

Auto-ViML Automatically Build Variant Interpretable ML models fast! Auto_ViML is pronounced "auto vimal" (autovimal logo created by Sanket Ghanmare) N

397 Dec 30, 2022