A novel framework to automatically learn high-quality scanning of non-planar, complex anisotropic appearance.

Overview

appearance-scanner

About

This repository is an implementation of the neural network proposed in Free-form Scanning of Non-planar Appearance with Neural Trace Photography

For any questions, please email xiaohema98 at gmail.com

Usage

System Requirement

  • Windows or Linux(The codes are validated on Win10, Ubuntu 18.04 and Ubuntu 16.04)
  • Python >= 3.6.0
  • Pytorch >= 1.6.0
  • tensorflow>=1.11.0, meshlab and matlab are needed if you process the test data we provide

Training

  1. move to appearance_scanner
  2. run train.bat or train.sh according to your own platform

Notice that the data generation step

python data_utils/origin_parameter_generator_n2d.py %data_root% %Sample_num% %train_ratio%

should be run only once.

Training Visulization

When training is started, you can open tensorboard to observe the training process. There will be two log images of a certain training sample, one is the sampled lumitexels from 64 views and the other is an composite image from six images in the order of groundtruth lumitexel, groundtruth diffuse lumitexel, groundtruth specular lumitexel, predicted lumitexel, predicted diffuse lumitexel and predicted specular lumitexel.

Trained lighting pattern will also be showed. Trained model will be found in the log_dir set in train.bat/train.sh.

License

Our source code is released under the GPL-3.0 license for acadmic purposes. The only requirement for using the code in your research is to cite our paper:

@article{Ma:2021:Scanner,
author = {Ma, Xiaohe and Kang, Kaizhang and Zhu, Ruisheng and Wu, Hongzhi and Zhou, Kun},
title = {Free-Form Scanning of Non-Planar Appearance with Neural Trace Photography},
year = {2021},
issue_date = {August 2021},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
volume = {40},
number = {4},
issn = {0730-0301},
url = {https://doi.org/10.1145/3450626.3459679},
doi = {10.1145/3450626.3459679},
journal = {ACM Trans. Graph.},
month = jul,
articleno = {124},
numpages = {13},
keywords = {illumination multiplexing, SVBRDF, optimal lighting pattern}
}

For commercial licensing options, please email hwu at acm.org. See COPYING for the open source license.

Reconstruction process

The reconstruction needs photographs taken with our scanner, a pre-trained network model and a pre-captured geometry shape as input. First, perform structure-from-motion with COLMAP, resulting in a 3D point cloud and camera poses with respect to it. Next, this point cloud is precisely aligned with the pre-captured shape. Then the view information of each vertex can be assembled as the input of the network. Last, we fit the predicted grayscale specular lumitexel with L-BFGS-B, to obtain the refletance parameters.

Download our Cheongsam test data and unzip it in appearance_scanner/data/.

Three sample photographs captured from the Cheongsam object. The brightness of the original images has been doubled for a better visualization.

Download our model and unzip it in appearance_scanner/.

1. Camera Registration

1.1 Run SFM/run.bat first to brighten the raw images

1.2 Open Colmap and do the following steps

1.2.1 New project

1.2.2 Feature extraction

Copy the parameters of our camera in device_configuration/cam.txt to Custom parameters.

1.2.3 Feature matching

Tick guided_matching and run.

1.2.4 Reconstruction options

Do not tick multiple_models in the General sheet.

Do not tick refine_focal_length/refine_extra_params/use_pba in the Bundle sheet.

Start reconstruction.

1.2.5 Bundle adjustment

Do not tick refine_focal_length/refine_principal_point/refine_extra_params.

1.2.6

Make a folder named undistort_feature in Cheongsam/ and export model as text in undistort_feature folder. Three files including cameras.txt, images.txt and point3D.txt will be saved.

1.2.7

Dense reconstruction -> select undistort_feature folder -> Undistortion -> Stereo

Since we upload all the photos we taken, it will take a long time to run this step. We strongly recommend you run

colmap stereo_fusion --workspace_path path --input_type photometric --output_path path/fused.ply

//change path to undistort_feature folder

when the files amount in undistort_feature/stereo/normal_maps arise to around 200-250. It will output a coarse point cloud in undistort_feature/ .

Delete the noise points and the table plane.

Save fused.ply.

2. Extract measurements

move your own model to models/ and run appearance_scanner/test_files/prepare_pattern.bat

run extract_measurements/run.bat

3. Align mesh

3.1 Use meshlab to align mesh roughly

Open fused.ply and Cheongsam/scan/Cheongsam.ply in the same meshlab window. Cheongsam.ply is pre-capptured with a commercial mobile 3D scanner, EinScan Pro 2X Plus.

Align two mesh and save project file in Cheongsam/scan/Cheongsam.aln, which records the transform matrix between two meshes.

run CoherentPointDrift/run.bat to align Cheongsam.ply to fused.ply.

3.2 Further Alignment

run CoherentPointDrift/CoherentPointDrift-master/simplify/run.bat to simplify two meshes. It will call meshlabserver to simplify two meshes so that save the processing time.

Open the CPD project in Matlab and run main.m.

After alignment done, run CoherentPointDrift/run_pass2.bat. meshed-poisson_obj.ply will be saved in undistort_feature/ .

You should open fused.ply and meshed-poisson_obj.ply in the same meshlab window to check the quality of alignment. It is a key factor in the final result.

4. Generate view information from registrated cameras

4.1 Remesh

run ACVD/aarun.bat

save undistort_feature/meshed-poisson_obj_remeshed.ply as undistort_feature/meshed-poisson_obj_remeshed.obj

It is not necessary to reconstruct all the vertices on the pre-captured shape in our case. The remesh step will output an optimized 3D triangular mesh with a user defined vertex budget, which is controlled by NVERTICES in aarun.bat.

4.2 uvatlas

copy data_processing/device_configuration/extrinsic.bin to undistort_feature/ copy Cheongsam/512.exr and 1024.exr to undistort_feature/

run generate_texture/trans.bat to transform mesh from colmap frame to world frame in our system and generate uv maps.

We recommend that you generate uv maps with resolution of 512x512 because it will save a lot of time and retain most details. The resolution of the results in our paper is 1024x1024.

You can set UVMAP_WIDTH and UVMAP_HEIGHT to 1024 in uv/uv_generator.bat if you pursue higher quality.

4.3 Compute view information

Downloads embree and copy bin/embree3.dll, glfw3.dll, tbb12.dll to generate_texture/.

Downloads opencv and copy opencv_world#v.dll to generate_texture/. We use opencv3.4.3 in our project.

in generate_texture/texgen.bat, set TEXTURE_RESOLUTION to the certain resolution

choose the same line or the other reference on meshed-poisson_obj_remeshed.obj and on the physical object, then meature the lengths of both. Set the results to COLMAP_L and REAL_L. REAL_L in mm.

The marker cylinder's diameter is 10cm, so we set REAL_L to 100.

run generate_texture/texgen.bat to output view information of all registrated cameras.

5. Gather data

run gather_data/run.bat to gather the inputs to the network for each valid pixel on the texture map. A folder named images_{resolution} will be made in Cheongsam/.

6. Fitting

  1. Change %DATA_ROOT% and %TEXTURE_MAP_SIZE% in fitting/tf_ggx_render/run.bat. Then run fitting/tf_ggx_render/run.bat.
  2. A folder named fitting_folder_for_server will be generated under texture_{resolution}.
  3. Upload the entire folder generated in previous step to a linux server.
  4. Change current path of terminal to fitting_folder_for_server\fitting_temp\tf_ggx_render, then run split.sh or split1024.sh according to the resolution you chosen. (split.sh is for 512. If you want to use custom texture map resolution, you may need to modify the $TEX_RESOLUTION in split.sh)
  5. When the fitting procedure finished, a folder named Cheongsam/images_{resolution}/data_for_server/data/images/data_for_server/fitted_grey will be generated. It contains the final texture maps, including normal_fitted_global.exr, tangent_fitted_global.exr, axay_fitted.exr, pd_fitted.exr and ps_fitted.exr.
    Note: If you find the split.sh cannot run properly and complain about abscent which_server argument, it's probably caused by the difference of linux and windows. Reading in the sh file and writing it with no changing of content on sever can fix this issue.
diffuse specular roughness
normal tangent

7. Render results

We use the anisotropic GGX model to represent reflectance. The object can be rendered with path tracing using NVIDIA OptiX or openGL.

Reference & Third party tools

Shining3D. 2021. EinScan Pro 2X Plus Handheld Industrial Scanner. Retrieved January, 2021 from https://www.einscan.com/handheld-3d-scanner/2x-plus/

Colmap: https://demuc.de/colmap/

Coherent Point Drift: https://ieeexplore.ieee.org/document/5432191

ACVD: https://github.com/valette/ACVD

Embree: https://www.embree.org/

OpenCV: https://opencv.org/

You might also like...
Python and C++ implementation of
Python and C++ implementation of "MarkerPose: Robust real-time planar target tracking for accurate stereo pose estimation". Accepted at LXCV @ CVPR 2021.

MarkerPose: Robust real-time planar target tracking for accurate stereo pose estimation This is a PyTorch and LibTorch implementation of MarkerPose: a

This package is for running the semantic SLAM algorithm using extracted planar surfaces from the received detection
This package is for running the semantic SLAM algorithm using extracted planar surfaces from the received detection

Semantic SLAM This package can perform optimization of pose estimated from VO/VIO methods which tend to drift over time. It uses planar surfaces extra

A Planar RGB-D SLAM which utilizes Manhattan World structure to provide optimal camera pose trajectory while also providing a sparse reconstruction containing points, lines and planes, and a dense surfel-based reconstruction.
A Planar RGB-D SLAM which utilizes Manhattan World structure to provide optimal camera pose trajectory while also providing a sparse reconstruction containing points, lines and planes, and a dense surfel-based reconstruction.

ManhattanSLAM Authors: Raza Yunus, Yanyan Li and Federico Tombari ManhattanSLAM is a real-time SLAM library for RGB-D cameras that computes the camera

Indoor Panorama Planar 3D Reconstruction via Divide and Conquer
Indoor Panorama Planar 3D Reconstruction via Divide and Conquer

HV-plane reconstruction from a single 360 image Code for our paper in CVPR 2021: Indoor Panorama Planar 3D Reconstruction via Divide and Conquer (pape

[ICCV 2021 (oral)] Planar Surface Reconstruction from Sparse Views
[ICCV 2021 (oral)] Planar Surface Reconstruction from Sparse Views

Planar Surface Reconstruction From Sparse Views Linyi Jin, Shengyi Qian, Andrew Owens, David F. Fouhey University of Michigan ICCV 2021 (Oral) This re

PyTorch implementation of HDN(Homography Decomposition Networks) for planar object tracking
PyTorch implementation of HDN(Homography Decomposition Networks) for planar object tracking

Homography Decomposition Networks for Planar Object Tracking This project is the offical PyTorch implementation of HDN(Homography Decomposition Networ

Pytorch implementation of paper:
Pytorch implementation of paper: "NeurMiPs: Neural Mixture of Planar Experts for View Synthesis"

NeurMips: Neural Mixture of Planar Experts for View Synthesis This is the official repo for PyTorch implementation of paper "NeurMips: Neural Mixture

Asterisk is a framework to generate high-quality training datasets at scale
Asterisk is a framework to generate high-quality training datasets at scale

Asterisk is a framework to generate high-quality training datasets at scale

 DeepFaceEditing: Deep Face Generation and Editing with Disentangled Geometry and Appearance Control
DeepFaceEditing: Deep Face Generation and Editing with Disentangled Geometry and Appearance Control

DeepFaceEditing: Deep Face Generation and Editing with Disentangled Geometry and Appearance Control One version of our system is implemented using the

Owner
Xiaohe Ma
Xiaohe Ma
[MICCAI'20] AlignShift: Bridging the Gap of Imaging Thickness in 3D Anisotropic Volumes

AlignShift NEW: Code for our new MICCAI'21 paper "Asymmetric 3D Context Fusion for Universal Lesion Detection" will also be pushed to this repository

Medical 3D Vision 42 Jan 6, 2023
ZSL-KG is a general-purpose zero-shot learning framework with a novel transformer graph convolutional network (TrGCN) to learn class representation from common sense knowledge graphs.

ZSL-KG is a general-purpose zero-shot learning framework with a novel transformer graph convolutional network (TrGCN) to learn class representa

Bats Research 94 Nov 21, 2022
Complex-Valued Neural Networks (CVNN)Complex-Valued Neural Networks (CVNN)

Complex-Valued Neural Networks (CVNN) Done by @NEGU93 - J. Agustin Barrachina Using this library, the only difference with a Tensorflow code is that y

youceF 1 Nov 12, 2021
Automatic Calibration for Non-repetitive Scanning Solid-State LiDAR and Camera Systems

ACSC Automatic extrinsic calibration for non-repetitive scanning solid-state LiDAR and camera systems. System Architecture 1. Dependency Tested with U

KINO 192 Dec 13, 2022
Unified tracking framework with a single appearance model

Paper: Do different tracking tasks require different appearance model? [ArXiv] (comming soon) [Project Page] (comming soon) UniTrack is a simple and U

ZhongdaoWang 300 Dec 24, 2022
Open source repository for the code accompanying the paper 'Non-Rigid Neural Radiance Fields Reconstruction and Novel View Synthesis of a Deforming Scene from Monocular Video'.

Non-Rigid Neural Radiance Fields This is the official repository for the project "Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synt

Facebook Research 296 Dec 29, 2022
The code for the CVPR 2021 paper Neural Deformation Graphs, a novel approach for globally-consistent deformation tracking and 3D reconstruction of non-rigid objects.

Neural Deformation Graphs Project Page | Paper | Video Neural Deformation Graphs for Globally-consistent Non-rigid Reconstruction Aljaž Božič, Pablo P

Aljaz Bozic 134 Dec 16, 2022
E2EC: An End-to-End Contour-based Method for High-Quality High-Speed Instance Segmentation

E2EC: An End-to-End Contour-based Method for High-Quality High-Speed Instance Segmentation E2EC: An End-to-End Contour-based Method for High-Quality H

zhangtao 146 Dec 29, 2022
git《Learning Pairwise Inter-Plane Relations for Piecewise Planar Reconstruction》(ECCV 2020) GitHub:

Learning Pairwise Inter-Plane Relations for Piecewise Planar Reconstruction Code for the ECCV 2020 paper by Yiming Qian and Yasutaka Furukawa Getting

null 37 Dec 4, 2022
Planar Prior Assisted PatchMatch Multi-View Stereo

ACMP [News] The code for ACMH is released!!! [News] The code for ACMM is released!!! About This repository contains the code for the paper Planar Prio

Qingshan Xu 127 Dec 31, 2022