Code for "NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video", CVPR 2021 oral

Last update: Dec 30, 2022

Overview

NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video

Project Page | Paper

NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video
Jiaming Sun^*, Yiming Xie^*, Linghao Chen, Xiaowei Zhou, Hujun Bao
CVPR 2021 (Oral Presentation)

Code release ETA

We plan to release the code within a month, stay tuned. Please subscribe to this discussion thread if you wish to be notified of the code release. In the meanwhile, discussions about the paper are welcomed in the discussion panel.

Citation

If you find this code useful for your research, please use the following BibTeX entry.

@article{sun2021neucon,
  title={{NeuralRecon}: Real-Time Coherent {3D} Reconstruction from Monocular Video},
  author={Sun, Jiaming and Xie, Yiming and Chen, Linghao and Zhou, Xiaowei and Bao, Hujun},
  journal={CVPR},
  year={2021}
}

Copyright

This work is affiliated with ZJU-SenseTime Joint Lab of 3D Vision, and its intellectual property belongs to SenseTime Group Ltd.

Copyright SenseTime. All Rights Reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

Comments

about other datasets and data process of poses

https://github.com/zju3dv/NeuralRecon/issues/76#issuecomment-1059915310 could you please give the result on 7-scenes or how to process the pose ,in fact, i have tried it on several datasets,some need to add a 1.5 ,some need to rot the x-axis like process the demo data in ARkit , but all of the result seems worse than your demo data ,the demo result is all right,so i think my operate is right, the problem in processing the pose seems with the biggest probability。（i'm not very good at english,and i'm new in 3D Reconstruction,so Please forgive my question may not be standardized） here are some results in other datasets first ,the demo data provided by the author $9K5XMZ{QO1G_B3L6WLJ`W K$

the TUM dateset

the scannet data(i'm not able to download the full scannet,so it's 100 times downsampled,it seems will be ok if use more photos) $)8(GK$(0{O4_EI(_D2PT9YR$ the 7-scenes dataset finally,i use my phone get a video ,and use the orb-slam2 to get the pose of key-frame(i can't use a iphone to use ARkit)

opened by neuhsm 13

ScanNet data

Q1: As described in ScanNet, the data structure of ScanNet is:

<scanId>
|-- <scanId>.sens
    RGB-D sensor stream containing color frames, depth frames, camera poses and other data
|-- <scanId>_vh_clean.ply
    High quality reconstructed mesh
|-- <scanId>_vh_clean_2.ply
    Cleaned and decimated mesh for semantic annotations
|-- <scanId>_vh_clean_2.0.010000.segs.json
    Over-segmentation of annotation mesh
|-- <scanId>.aggregation.json, <scanId>_vh_clean.aggregation.json
    Aggregated instance-level semantic annotations on lo-res, hi-res meshes, respectively
|-- <scanId>_vh_clean_2.0.010000.segs.json, <scanId>_vh_clean.segs.json
    Over-segmentation of lo-res, hi-res meshes, respectively (referenced by aggregated semantic annotations)
|-- <scanId>_vh_clean_2.labels.ply
    Visualization of aggregated semantic segmentation; colored by nyu40 labels (see img/legend; ply property 'label' denotes the ScanNet label id)
|-- <scanId>_2d-label.zip
    Raw 2d projections of aggregated annotation labels as 16-bit pngs with ScanNet label ids
|-- <scanId>_2d-instance.zip
    Raw 2d projections of aggregated annotation instances as 8-bit pngs
|-- <scanId>_2d-label-filt.zip
    Filtered 2d projections of aggregated annotation labels as 16-bit pngs with ScanNet label ids
|-- <scanId>_2d-instance-filt.zip
    Filtered 2d projections of aggregated annotation instances as 8-bit pngs

However, you mention that the data structure is:

DATAROOT
└───scannet
│   └───scans
│   |   └───scene0000_00
│   |       └───color
│   |       │   │   0.jpg
│   |       │   │   1.jpg
│   |       │   │   ...
│   |       │   ...
│   └───scans_test
│   |   └───scene0707_00
│   |       └───color
│   |       │   │   0.jpg
│   |       │   │   1.jpg
│   |       │   │   ...
│   |       │   ...
|   └───scannetv2_test.txt
|   └───scannetv2_train.txt
|   └───scannetv2_val.txt

So, how should I modify the original ScanNet data to satisfy the requirements?

Q2: Following the guidance, I reconstruct the mesh from the demo video. Why does the mesh have no color?

opened by NoOneUST 13

problem in running generate_gt.py,

(py379) peter@DESKTOP-A5F108R:~/code/NeuralRecon/NeuralRecon-master$ python tools/tsdf_fusion/generate_gt.py --data_path /home/peter/code/NeuralRecon/scannet --save_name all_tsdf_9 --window_size 9 Traceback (most recent call last): File "tools/tsdf_fusion/generate_gt.py", line 275, in files = split_list(files, all_proc) File "tools/tsdf_fusion/generate_gt.py", line 231, in split_list assert len(_list) >= n AssertionError

opened by guaiweidou12315 11
"No valid points" when running demo on example data

Hi Team, thanks for sharing this outstanding piece of work. When trying to run demo.py on the example data provided in DEMO.md I come across the error message below after following all the steps. Any idea why reconstruction is failing? I get the same error when I run it on my own captured ARKit data.

2021-05-10 19:51:55.606 | INFO | main::35 - Found SyncedPoses.txt, skipping data pre-processing... 2021-05-10 19:51:55.606 | INFO | main::37 - Running NeuralRecon... 2021-05-10 19:51:55.613 | INFO | main::50 - Initializing the model on GPU... 2021-05-10 19:51:58.977 | INFO | main::58 - Resuming from ./checkpoints/model_000047.ckpt 2021-05-10 19:51:59.299 | INFO | main::64 - Start inference.. 0%| | 0/111 [00:00<?, ?it/s]2021-05-10 19:52:00.083 | WARNING | models.neucon_network:forward:184 - no valid points: scale 1 0%| | 0/111 [00:00<?, ?it/s] Traceback (most recent call last): File "demo.py", line 78, in save_mesh_scene(outputs, sample, epoch_idx) File "NeuralRecon/utils.py", line 320, in call self.tsdf_volume.append(outputs['scene_tsdf']) KeyError: 'scene_tsdf'

opened by ftasse 11
AttributeError: module 'torchsparse_backend' has no attribute 'hash_forward'

Hello,I'm sorry to have taken so long to get back to you.Recently,I've installed the system and configurate environment again,but this error is still display(I don't know how to reopen the past issue,so I create a new one.I feel sorry about that)： 2021-06-05 15:28:46.856 | INFO | main::35 - Found SyncedPoses.txt, skipping data pre-processing... 2021-06-05 15:28:46.856 | INFO | main::37 - Running NeuralRecon... 2021-06-05 15:28:46.874 | INFO | main::50 - Initializing the model on GPU... 2021-06-05 15:28:50.821 | INFO | main::58 - Resuming from ./checkpoints/model_000047.ckpt 2021-06-05 15:28:51.211 | INFO | main::64 - Start inference.. 0%| | 0/110 [00:00<?, ?it/s]/home/u/Documents/NeuralRecon-master/ops/back_project.py:29: UserWarning: This overload of nonzero is deprecated: nonzero(Tensor input, *, Tensor out) Consider using one of the following signatures instead: nonzero(Tensor input, *, bool as_tuple) (Triggered internally at /opt/conda/conda-bld/pytorch_1595629427478/work/torch/csrc/utils/python_arg_parser.cpp:766.) batch_ind = torch.nonzero(coords[:, 0] == batch).squeeze(1) 0%| | 0/110 [00:00<?, ?it/s] Traceback (most recent call last): File "demo.py", line 76, in outputs, loss_dict = model(sample, save_scene) File "/home/u/anaconda3/envs/neucon/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(*input, **kwargs) File "/home/u/anaconda3/envs/neucon/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 153, in forward return self.module(*inputs[0], **kwargs[0]) File "/home/u/anaconda3/envs/neucon/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(*input, **kwargs) File "/home/u/Documents/NeuralRecon-master/models/neuralrecon.py", line 82, in forward outputs, loss_dict = self.neucon_net(features, inputs, outputs) File "/home/u/anaconda3/envs/neucon/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(*input, **kwargs) File "/home/u/Documents/NeuralRecon-master/models/neucon_network.py", line 157, in forward feat = self.sp_convsi File "/home/u/anaconda3/envs/neucon/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(*input, **kwargs) File "/home/u/Documents/NeuralRecon-master/models/modules.py", line 147, in forward x0 = initial_voxelize(z, self.pres, self.vres) File "/home/u/Documents/NeuralRecon-master/ops/torchsparse_utils.py", line 19, in initial_voxelize pc_hash = spf.sphash(torch.floor(new_float_coord).int()) File "/home/u/anaconda3/envs/neucon/lib/python3.7/site-packages/torchsparse/nn/functional/hash.py", line 39, in sphash return HashGPU.apply(idx) File "/home/u/anaconda3/envs/neucon/lib/python3.7/site-packages/torchsparse/nn/functional/hash.py", line 11, in forward return torchsparse_backend.hash_forward(idx.contiguous()) AttributeError: module 'torchsparse_backend' has no attribute 'hash_forward'

I'm sure torchsparse is correctly installed and compilation by the end of the installation is succeeded.Is this casused by a mismatch between the torch version and the torchsparse version? my pytorch = 1.6.0;torchvision == 0.7.0

opened by Nekomiya-fubuki 9
using NeuralRecon in a realtime camera

Hello, is there a way to use NeuralRecon in my camera input (using Unity)? I would highly appreciate it if you could share a collab on how to use NeuralRecons API (how to pass images?).

I was working to work with Atlas but I just found out about you today and was hoping I could make my research work with NeuralRecon to show what your research is capable of!

opened by Ademord 9
no valid points: scale 0 when runing example data

the following is the detailed error infos, i search the similar issues and find #6 , but i check the code and the X-Y plane in the ARKit coordinates has be moved downwards a little in tools/process_arkit_data.py(cam_pose_dict[id][2, 3] += 1.5). so i wonder to know what's wrong with it?

@JiamingSuen

2021-08-07 10:21:25.395 | INFO | main::32 - First run on this captured data, start the pre-processing... Extract images from video... 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5301/5301 [00:39<00:00, 135.46it/s] Load intrinsics and extrinsics Processing camera intrinsics...: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 5301/5301 [00:00<00:00, 367765.61it/s] data source: ARKit Saving camera extrinsics...: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████| 5301/5301 [00:00<00:00, 7865.03it/s] Saving camera intrinsics...: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████| 5301/5301 [00:00<00:00, 8375.12it/s] Keyframes selection...: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████| 5301/5301 [00:00<00:00, 37386.61it/s] Saving fragments file...: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 110/110 [00:00<00:00, 148830.14it/s] 2021-08-07 10:22:06.263 | INFO | main::37 - Running NeuralRecon... 2021-08-07 10:22:06.265 | INFO | main::50 - Initializing the model on GPU... 2021-08-07 10:22:07.920 | INFO | main::58 - Resuming from ./checkpoints/release/model_000047.ckpt 2021-08-07 10:22:08.043 | INFO | main::64 - Start inference.. 0%| | 0/110 [00:00<?, ?it/s]/home/achao/3d/debug-NeuralRecon/NeuralRecon/ops/back_project.py:29: UserWarning: This overload of nonzero is deprecated: nonzero(Tensor input, *, Tensor out) Consider using one of the following signatures instead: nonzero(Tensor input, *, bool as_tuple) (Triggered internally at /pytorch/torch/csrc/utils/python_arg_parser.cpp:766.) batch_ind = torch.nonzero(coords[:, 0] == batch).squeeze(1) 2021-08-07 10:22:08.267 | WARNING | models.neucon_network:forward:184 - no valid points: scale 0 0%| | 0/110 [00:00<?, ?it/s] Traceback (most recent call last): File "demo.py", line 89, in save_mesh_scene.save_incremental(epoch_idx, 0, sample['imgs'][0], outputs) File "/home/achao/3d/debug-NeuralRecon/NeuralRecon/utils.py", line 256, in save_incremental tsdf_volume = outputs['scene_tsdf'][batch_idx].data.cpu().numpy() KeyError: 'scene_tsdf'

opened by achao2013 7
Network trained with 4 GPUs and 54 epochs has worse performance

I trained Neucon on 4 RTX5000 (16G) for 54 epochs. However, the evaluations on the Scannetdataset test split are worse than your released model trained for only 47 epochs.

Here are the results from my trained model (54 epochs) : AbsRel 0.072 AbsDiff 0.117 SqRel 0.037 RMSE 0.207 LogRMSE 0.120 r1 0.927 r2 0.960 r3 0.973 complete 0.861 dist1 0.066 dist2 0.179 prec 0.568 recal 0.349 fscore 0.429

Here are the results from your released model (47 epochs): AbsRel 0.065 AbsDiff 0.098 SqRel 0.038 RMSE 0.197 LogRMSE 0.113 r1 0.933 r2 0.961 r3 0.974 complete 0.892 dist1 0.053 dist2 0.133 prec 0.687 recal 0.473 fscore 0.558

Do you have any clues for this? Thanks a lot!

opened by ZuoJiaxing 5
Some questions about training requirement and convergence time

Hi, friends, as much as I can see from the demo, your work is really great. But I have some questions that I want you to answer. The problem is that I don't have enough GPUs, all I have is a single RTX2080Ti or RTX3090, I really want to konw your training consumption both in graphic memory and time. Can I use 2080Ti to train your model even if it cost more time? And if RTX3090 supports your code ? Thank you very much , I 'll be glad to receive your earliest reply when you are not too busy.

opened by Radium98 5
texture mapping on your output.ply

could you please tell me, how do you generate this textures on your gray output.ply file. Like this two example you offered. And how do you build your AR demo? On own data captured by iphone.

opened by cucdengjunli 4
Apply to new datasets

Hi,

Thank you for the great work.

I try to apply demo.py on TUM datasets and EuRoc(drone) datasets. But I didn’t get an accurate result.

The first picture is the result of applying the TUM dataset picture, and the second image is the result of using EuRoc data.

Is it correct that the results are like this? Have you applied the NeuralRecon code to the datasets mentioned?

Thanks, Hyunjin Kim.

opened by hyunJIN7 4
Error when trying to convert to onnx

Hi team:

Has anyone tried to convert the NeuralRecon model to onnx format? I remove the DataParallel before conversion (the DataParallel is reported to cause onnx exportion problem) but still facing errors like: "RuntimeError: 0 INTERNAL ASSERT FAILED at "/opt/conda/conda-bld/pytorch_1656352657443/work/torch/csrc/jit/ir/alias_analysis.cpp":608, please report a bug to PyTorch. We don't have an op for aten::eq but it isn't a special case. Argument types: Tensor, bool,"

opened by scutzetao 0
Docker file to overcome pycuda issue

Discussed in https://github.com/zju3dv/NeuralRecon/discussions/123

^{Originally posted by Viditagarwal7479 December 4, 2022} Hey,

While setting up the repository, I am facing issues with setting up the environment due to pycuda issues.

I looked on the web for solutions and also tried building pycuda from the source still it is failing.

Can someone whose environment has been properly dockerize it and make it available publically, as many other users are facing pycuda issue and are not able to overcome it?

Thanks Vidit Agarwal

opened by Viditagarwal7479 0

a question about transforms code

Thank to the nice work！But I have some questions about the transforms I just wondering why origin[2] is the const equal to -0.2 , so any vol_origin_parital's z is the const? Isn't the origin of each fragment different? Why is the point of the z-axis constant?

-------adjust volume bounds-------

    num_layers = 3
    center = (torch.tensor(((bnds[0, 1] + bnds[0, 0]) / 2, (bnds[1, 1] + bnds[1, 0]) / 2, -0.2)) - data[       
        'vol_origin']) / self.voxel_size
    center[:2] = torch.round(center[:2] / 2 ** num_layers) * 2 ** num_layers
    center[2] = torch.floor(center[2] / 2 ** num_layers) * 2 ** num_layers
    origin = torch.zeros_like(center)
    origin[:2] = center[:2] - torch.tensor(self.voxel_dim[:2]) // 2
    origin[2] = center[2]
    vol_origin_partial = origin * self.voxel_size + data['vol_origin']

opened by MarklearnML 0

pycuda

pycuda.driver.CompileError: nvcc compilation of /tmp/tmp3sh8k726/kernel.cu failed [command: /usr/local/cuda-11.0/bin/nvcc --cubin -arch sm_86 -I/home/dell/anaconda3/envs/neucon/lib/python3.7/site-packages/pycuda/cuda kernel.cu] [stderr: nvcc fatal : Value 'sm_86' is not defined for option 'gpu-architecture' ]

opened by yuyingliu717 0

Project dependencies may have API risk issues

Hi, In NeuralRecon, inappropriate dependency versioning constraints can cause risks.

Below are the dependencies and version constraints that the project is using

ray
transforms3d
tqdm
opencv-python
numba
pycuda
tensorboardX
scikit-image
trimesh~=3.9.18
yacs
h5py
loguru
gdown
pyrender~=0.1.45
pyglet
open3d

The version constraint == will introduce the risk of dependency conflicts because the scope of dependencies is too strict. The version constraint No Upper Bound and * will introduce the risk of the missing API Error because the latest version of the dependencies may remove some APIs.

After further analysis, in this project, The version constraint of dependency tqdm can be changed to >=4.36.0,<=4.64.0. The version constraint of dependency numba can be changed to >=0.34.0,<=0.55.2. The version constraint of dependency pycuda can be changed to >=2015.1,<=2021.1. The version constraint of dependency pyglet can be changed to >=1.3.0rc2,<=1.4.11.

The above modification suggestions can reduce the dependency conflicts as much as possible, and introduce the latest version as much as possible without calling Error in the projects.

The invocation of the current project includes all the following methods.

The calling methods from the tqdm

tqdm.tqdm

The calling methods from the numba

numba.prange
numba.njit

The calling methods from the pycuda

pycuda.compiler.SourceModule

The calling methods from the pyglet

datetime.datetime.now

The calling methods from the all methods

numpy.concatenate.append
tools.visualize_metrics.visualize
torchsparse.PointTensor
numpy.sin
torch.cuda.empty_cache
coords_world_s.abs.squeeze
occ_target.view.sum
f.writelines
numpy.ones
train_sample
mask.unsqueeze.permute
grid.unsqueeze.cuda.float
load_camera_intrinsic
self.upsample
cam_pose.float.to
yv.reshape
utils.sparse_to_dense_channel
self._vol_dim.torch.ones.to
torch.rand
args
enumerate
test_sample
up_feat.view.view
self.inner2
coords.detach.clone.long
Renderer.mesh_opengl
numpy.random.choice
x.split
tqdm.tqdm
utils.apply_log_transform.sum
outputs.append
BasicConvolutionBlock
extrinsics_list.append
z.C.view
up_coords.torch.cat.permute
torch.cat.detach
ray.get
z.idx_query.get
self.convq
torch.multiprocessing.set_sharing_strategy
inserted_coords.torch.round.int
utils.SaveScene
r_coords.shape.torch.zeros.to
grid.view.view
img.data.cpu.numpy
torch.Tensor.view.long.torch.Tensor.PointTensor.cuda
range
np.minimum
x.split.split
im_grid.view.view
mask.sum.sum
torch.stack
torch.zeros_like
numpy.logical_and
rs_grid.permute.contiguous
utils.coordinates.type
self.DemoDataset.super.__init__
features.permute.contiguous
self.transforms
numpy.abs
coords.shape.torch.zeros.cuda
vars.data.item
i.coords.torch.nonzero.squeeze
l.tsdf_vol_list.get_mesh
torch.nn.parallel.DistributedDataParallel
self.relu
global_coords_target.dim.global_coords_target.all
os.path.join
torch.nn.init.constant_
load_camera_pose.items
self.stage2
ResidualBlock
im_z.sum.unsqueeze.sum
self.fix_pose
full_tsdf_list.append
torch.ones_like
cv2.destroyAllWindows
pre_coords.unsqueeze
torch.randperm
self.read_scene_volumes
depth_im.float.to
self.reset
scale.self.global_volume.F.squeeze
float
torch.optim.Adam.state_dict
utils.tocuda
meshwrite
pyrender.Mesh.from_trimesh
rotx.dot
self.transform.pop
color_im.reshape.astype
cfg.defrost
np.loadtxt
numpy.stack
int
depth.append
self.scene.remove_node
trimesh.load
distances.append
global_volume.abs.any
torch.tensor
num.pre_coords.unsqueeze.repeat.contiguous
coords_world_s.T.view
np.maximum
xv.reshape
importlib.import_module
utils.apply_log_transform
torch.abs
self.fuse_to_global.keys
self.backbone2d
backbone.MnasMulti
self._cuda_integrate
torch.sign
open3d.pipelines.integration.ScalableTSDFVolume
torch.nn.DataParallel.train
self.occ_preds.append
in_scope_mask.permute.contiguous
r_coords.permute.contiguous
cam_pose.reshape.astype
args.__dict__.items
cv2.cvtColor
torch.from_numpy
print
torch.eye.inverse
torchvision.utils.make_grid
synchronize
l.tsdf_vol_list.get_volume
loguru.logger.add_image
l.tsdf_vol_list.integrate
self._voxel_size.self._vol_bnds.self._vol_bnds.np.round.copy
b.up_coords.torch.nonzero.squeeze
v.data.item
open3d.io.read_point_cloud
mask.view.view
self.tsdf_preds.append
numpy.meshgrid
torch.is_tensor
numpy.cross
datetime.datetime.now
torch.tensor.min
torch.nn.DataParallel
fn.endswith
zv.reshape.yv.reshape.xv.reshape.np.concatenate.astype
utils.sparse_to_dense_channel.any
i.self.tsdf_preds
middle_pose.inverse
self._vol_dim.np.zeros.astype
numpy.linalg.norm
vox_coords.astype.astype
load_camera_intrinsic.keys
neucon_network.NeuConNet
main
gru_fusion.GRUFusion
time.time
PIL.Image.open
preprocess
batch.coords.torch.nonzero.squeeze
batch.origin.unsqueeze.float
self.transform
numpy.floor
numpy.eye
open3d.geometry.PointCloud
torchsparse.nn.functional.spdevoxelize
torch.ones
depth_list.keys
new_input.items
b.pre_coords.torch.nonzero.squeeze
numpy.array.astype
cv2.resize
argparse.ArgumentParser.parse_args
p.data.nelement
self.transform.append
ray.init
transform
cx.cam_c.fx.cam_c.torch.round.long
id.str.self.scene.self.data_path.os.path.join.cv2.imread.astype
numpy.maximum.astype
self.stem
numba.njit
self.convr
utils.coordinates
depth_im.float.to.reshape
torch.Tensor
self.stage1
cfg.freeze
torch.distributed.is_available
z.C.int.view
ScanNetDataset
open3d.geometry.KDTreeFlann
sys.path.append
path_parser
torch.nn.Sequential
torchsparse.nn.functional.spcount
batch_idx.outputs.data.cpu.numpy.all
depth_im.reshape.astype
torch.nonzero
target_volume.squeeze.abs
intr.astype.astype
cv2.resize.permute
torch.meshgrid
transforms3d.quaternions.axangle2quat
NameError
proj_matrices.append
numpy.load
nn_correspondance
cy.cam_c.fy.cam_c.torch.round.long
max
align_corners.coords_world_s.old_voxel_dim.tsdf_s.view.torch.nn.functional.grid_sample.squeeze.get_volume
batch.origin.unsqueeze
scene.replace
self.get_target
torch.cat.sum
torchsparse.tensor.PointTensor
idx_query.transpose.contiguous.transpose
np.savez_compressed
idx_query.transpose.contiguous.int
torch.Generator
coords_world_s.abs.squeeze.any
np.savetxt
open3d.io.write_triangle_mesh
vars.cuda
numpy.minimum
img.data.cpu
mask1.astype
self.cuda.InOut
self.sp_convs.append
zv.reshape
modules.ConvGRU
self._vol_dim.torch.prod.item
torch.max
json.load
yacs.config.CfgNode
im_grid.view.abs
self.MnasMulti.super.__init__
cam_intr.reshape.astype
train
dim.data.cpu.numpy.tolist
align_corners.coords_world_s.old_voxel_dim.tsdf_s.view.torch.nn.functional.grid_sample.squeeze
mask.sum.float
torchsparse.nn.utils.get_kernel_offsets
self.build_list
r_coords.permute.contiguous.permute
sorted.keys
self.cfg.BACKBONE2D.ARC.split
occupancy.sum.data.cpu
dim.data.cpu.numpy
cv2.imwrite
datasets.transforms.Compose.IntrinsicsPoseToProjection
torch.nn.functional.interpolate
self.cuda.memcpy_htod
datasets.transforms.Compose.RandomTransformSpace
f.readlines.split
grid.view.unsqueeze
pycuda.driver.mem_alloc
config.cfg.freeze
zv.flatten
numpy.zeros
self.scene.add
parse_args
vars.items
coords_world_s.T.view.view
world_c.transpose.world2cam.torch.matmul.transpose
cam_intr.float.to
torch.distributed.get_rank
z.additional_features.get
self.point_transforms
z.C.int
self.transform.keys
self.normalizer
self.out1
loss.backward
self.integrate_tsdf
ARKitDataset
torch.unique
vars.detach
torch.cat.permute
tensor2numpy
pickle.load
utils.coordinates.view
threshold.dist1.astype
self.renderer.render
torch.cat.unsqueeze
self.save_mesh
g.self.dataset.len.torch.randperm.tolist
self.viewer.render_lock.release
old_voxel_dim.torch.Tensor.view
torch.clamp
self.tsdf2mesh.export
dict
torch.nn.utils.clip_grad_norm_
updated_coords.detach.clone.float
str
torchvision.models.mnasnet1_0
utils.SaveScene.close
os.path.isdir
line.split
self.NeuConNet.super.__init__
np.array.np.array.T.last_pose.cam_pose.np.linalg.inv.sum
torch.floor
up_coords.detach.clone.float
trimesh.Trimesh
global_coords.dim.global_coords.all
torch.nn.Linear
zip
r_coords.permute.contiguous.float
numpy.empty
data.np.stack.transpose
batch_idx.outputs.data.cpu
self.rotate_view_to_align_xyplane
xyzrgb.astype
torch.cat.abs
torch.nn.DataParallel.state_dict
super
tools.process_arkit_data.process_data
np.linalg.inv
torch.cuda.manual_seed
torch.nn.ReLU
c.coords.shape.torch.zeros.cuda
batch_idx.outputs.data.cpu.numpy
torch.distributed.is_initialized
quat.quat.np.append.tolist
argparse.ArgumentParser.add_argument
SConv3d
mask.sum.unsqueeze
zv.flatten.yv.flatten.xv.flatten.torch.stack.long.to
len.items
cv2.VideoCapture.read
pyrender.Scene
vol_origin.astype.astype
self.pixel_std.type_as
loss_dict.update
all_ids.append
cv2.VideoCapture
numpy.append
np.array
skimage.measure.marching_cubes
torch.distributed.get_world_size
torch.matmul
torch.min
torch.cuda.memory_reserved
open.write
imgs.append
z.weights.get
self._cuda_src_mod.get_function
self.neucon_net
self._vol_dim.torch.zeros.to
config.cfg.TRAIN.LREPOCHS.split.split
ray_worker_ids.append
self._vol_dim.np.ones.astype
threshold.dist2.astype
torch.nn.DataParallel.parameters
numpy.log
rotx
coords.detach.clone
self.pixel_mean.type_as
tuple
self.ScanNetDataset.super.__init__
numpy.empty_like
rigid_transform
config.cfg.defrost
self.gru_fusion
sum
open3d.camera.PinholeCameraIntrinsic
pcd_trgt.voxel_down_sample.voxel_down_sample
self.convz
load_camera_intrinsic.items
self.cuda.memcpy_dtoh
voxel_dim.long
numpy.maximum
pre_coords.unsqueeze.repeat
torchsparse.nn.functional.spvoxelize
MVSDataset
relative_origin.cuda.long
pyrender.Viewer
numpy.ceil
self.ConvGRU.super.__init__
torch.optim.lr_scheduler.MultiStepLR
join
im_z.sum.unsqueeze.view
line_data_list.int.str.zfill
new_tensor.cmaps.setdefault
align_corners.coords_world_s.old_voxel_dim.tsdf_s.view.torch.nn.functional.grid_sample.squeeze.abs
filepath.cv2.imread.astype
cv2.VideoCapture.get
torch.load
numpy.loadtxt
idx_query.transpose.contiguous
os.path.exists
cfg.LOGDIR.split
np.linspace
numpy.linalg.inv
outputs.keys
ckpt_list.append
cfg.merge_from_list
self.modules
open3d.pipelines.integration.ScalableTSDFVolume.extract_triangle_mesh
torch.optim.Adam.zero_grad
torch.save
min_c.max_c.int.data.cpu.numpy
open
cam_line.split
loguru.logger.add
FileNotFoundError
self.fuse_to_global
world_c.transpose
datasets.transforms.Compose.ToTensor
target_volume.squeeze.squeeze
self.save_scene_eval
torch.cat
colors.astype.astype
str.split
torch.nn.functional.grid_sample
torch.distributed.init_process_group
i.self.occ_preds
scale.inputs.permute
vars.detach.cpu
min
numpy.repeat
up_coords.torch.cat.permute.contiguous.view
torch.utils.data.DataLoader
batch_ind.coords.long
cam_intrinsics.append
tools.tsdf_fusion.fusion.TSDFVolumeTorch
torchsparse.nn.functional.sphashquery
verts.np.round.astype
depth_all.update
scale.inputs.permute.contiguous
os.makedirs
numpy.round
self.out3
json.dump
split_list
x.s.idx_query.z.C.F.calc_ti_weights.transpose
utils.tensor2float
point_to_voxel
cfg.MODEL.PIXEL_MEAN.torch.Tensor.view
pyrender.MetallicRoughnessMaterial
self.downsample
num.pre_feat.unsqueeze.expand.contiguous
self._vol_origin.astype
cam_intr.float.to.float
torch.Tensor.view.long
NotImplementedError
vars.detach.cpu.numpy
loguru.logger.info
n_imgs.np.linspace.astype
ops.back_project.back_project
pcd_pred.voxel_down_sample.voxel_down_sample
tsdf_vol_list.append
initial_voxelize
updated_coords.detach.clone
self.viewer.close_external
mask.sum.view
BasicDeconvolutionBlock
math.ceil
nV.n_views.torch.ones.cuda
torch.nn.DataParallel.load_state_dict
torch.prod
torch.nn.functional.binary_cross_entropy_with_logits
occ.view.squeeze
x.s.z.C.torch.floor.int
self.scene.clear
self.save_mesh.append
tsdf_s.view
torch.Tensor.view
float.cam_info.np.array.reshape
load_camera_pose
os.path.dirname
process_with_single_worker.remote
cam_intr.float.to.reshape
vol_origin_partial.view
generate_pkl
os.listdir
argparse.ArgumentParser
scale.self.cfg.N_LAYER.self.cfg.N_VOX.torch.Tensor.cuda.int
coords.shape.torch.ones.to
min_c.max_c.int.data.cpu
open3d.geometry.RGBDImage.create_from_color_and_depth
save_fragment_pkl
np.amax
im.resize.resize
self.conv0
coords_batch.view.view
torch.unbind
datasets.find_dataset_def
extrinsics.data.cpu
x.s.idx_query.z.C.F.calc_ti_weights.transpose.contiguous
is_main_process
intrinsics.append
utils.DictAverageMeter
self.fusion_nets.append
utils.sparse_to_dense_channel.abs
tensor2float.items
self.cfg.N_VOX.torch.Tensor.cuda
torchsparse.cat
renderer
len
tsdf_vol.torch.zeros_like.bool
i.self.sp_convs
torch.cuda.set_device
occ_target.view.view
models.modules.SPVCNN
PIL.ImageOps.expand
self._voxel_size.self._vol_bnds.self._vol_bnds.np.round.copy.astype
utils.SaveScene.vis_incremental
k.metrics1.append
torch.mean
scale.self.fusion_nets
model
open3d.pipelines.integration.ScalableTSDFVolume.integrate
save_tsdf_full
torch.nn.BatchNorm1d
self.up2
im_z.im_z.mean
fragments.append
numpy.arccos
fragments.extend
numpy.cbrt
torch.full
self.up1
config.update_config
torchsparse.nn.Conv3d
numpy.newaxis.depth_pred.np.repeat.astype
torch.nn.DataParallel.eval
torchvision.models.MNASNet
min_c.max_c.int.data.cpu.numpy.tolist
self.viewer.render_lock.acquire
b.inputs.permute.contiguous
yv.flatten
align_corners.coords_world_s.old_voxel_dim.tsdf_s.view.torch.nn.functional.grid_sample.squeeze.integrate
torch.tanh
tensorboardX.SummaryWriter
self.net
data_path.split
format
origin.unsqueeze
xv.flatten
cam_pose.float.to.float
self.weight_initialization
numpy.prod
torch.norm
datasets.transforms.Compose.ResizeImage
batch_ind.up_coords.float
process
csv.reader
datasets.transforms.Compose.Compose
pyrender.IntrinsicsCamera
extract_frames
feats_batch.view.view
ops.generate_grids.generate_grid
os.path.isfile
_round_to_multiple_of
current_volume.abs.any
isinstance
self.save_mesh.index
torch.tensor.max
torch.arange
self.convert2dense
up_coords.detach.clone
loss_dict.items
occ_target.view.float
world_c.transpose.world2cam.torch.matmul.transpose.float
vars.detach.cpu.numpy.copy
self.compute_loss
torch.cat.view
torch.arange.flatten
f.readlines
numpy.hstack
super.__init__
torch.inverse
process_data
float.cam_info.np.array.reshape.transpose
torch.nn.ModuleList
numpy.savez_compressed
torch.Generator.manual_seed
coords_world_s.T.view.abs
gpu_mem_usage.append
test
utils.sparse_to_dense_torch
torch.stack.append
grid.unsqueeze.cuda
numpy.cos
Renderer
RuntimeError
cam_pose_all.update
tensor2float
torch.zeros
new_float_coord.torch.floor.int
cfg.merge_from_file
self._vol_bnds.copy.astype
cv2.imread
mask.unsqueeze.expand
numpy.array.reshape
open3d.utility.Vector3dVector
up_coords.torch.cat.permute.contiguous.append
getattr
dim.data.cpu
ops.generate_grids.generate_grid.detach
self.vis.vis_mesh
occ.squeeze.sum
occ.view.view
torchsparse.nn.ReLU
config.cfg.NeuralRecon.cuda
pycuda.driver.Device
torch.no_grad
sorted.items
metrics.update
numpy.nanmean
scale.self.global_volume.detach
torch.Tensor.cuda
numpy.vstack
torch.optim.Adam.step
visualize
pyrender.OffscreenRenderer
cam_pose.float.to.reshape
skimage.measure.marching_cubes_lewiner
numpy.dot
torch.distributed.barrier
pad_scannet
cam_poses.append
time.sleep
self.tsdf2mesh
loguru.logger.add_scalar
self.NeuralRecon.super.__init__
torch.manual_seed
models.NeuralRecon
self.inner1
torch.optim.Adam
numpy.zeros.fill
self.vis.close
relative_origin.cuda.long.cuda
all_bnds.append
numpy.max
datasets.sampler.DistributedSampler
voxel_to_point
numpy.sqrt
open.close
torch.nn.Dropout
loguru.logger.warning
self.dropout
origin.unsqueeze.float
self.vox2world
pre_feat.unsqueeze.expand
wrapper
pickle.dump
transforms3d.quaternions.quat2mat
torch.eye
self.update_map
np.zeros
torch.nn.DataParallel.cuda
self.conv2
grid_batch.unsqueeze.expand
feat.torch.ones_like.bool
tensor2numpy.items
torch.sigmoid
sorted
intrinsics_list.append
sync_intrinsics_and_poses
TSDFVolume
torch.optim.lr_scheduler.MultiStepLR.step
cfg.MODEL.PIXEL_STD.torch.Tensor.view
config.cfg.NeuralRecon.cuda.eval
self.out2
numba.prange
pycuda.compiler.SourceModule
utils.SaveScene.save_incremental
utils.apply_log_transform.view
self.renderer.delete
b.inputs.float
im_z.sum.unsqueeze
self._vol_dim.astype
np.amin
numpy.mean
datetime.datetime.now.strftime
depth_im.float.to.float
get_rank
origin_batch.float.unsqueeze
list
tools.evaluation_utils.eval_depth
ray.remote
up_coords.torch.cat.permute.contiguous
cv2.imshow
torch.nn.Conv2d
self.data.items
iter
old_origin.view.view
self.out_channels.append
np.linalg.norm
indices.append
torchsparse.nn.functional.calc_ti_weights
cv2.waitKey
utils.DictAverageMeter.mean
os.mkdir
grid_dim_z.grid_dim_y.grid_dim_x.np.array.astype
x.C.to
torchsparse.SparseTensor
numpy.array
lines.append
func
abs
poses.append
n.idx.ret.append
type
n_p.n_all.float
self.conv1
numpy.concatenate
torch.round
utils.DictAverageMeter.update
inputs.keys
self.read_depth
self.read_img
self._vol_bnds.copy
torchsparse.nn.functional.sphash
self.GRUFusion.super.__init__
kwargs.get
self.cam2pix
torchsparse.nn.BatchNorm
config.cfg.TRAIN.LREPOCHS.split
_get_depths
torch.log
np.arccos
ids.append
zv.flatten.yv.flatten.xv.flatten.torch.stack.long
numpy.asarray
target_volume.abs.target_volume.unsqueeze
numpy.stack.astype
up_coords.torch.cat.permute.contiguous.detach
rotation_matrix4x4.torch.from_numpy.float
save_mesh_scene
self.tsdf_cashe.keys
self.read_cam_file
open3d.geometry.Image
tools.evaluation_utils.eval_mesh
i.str.zfill
torch.device
utils.save_scalars
open3d.geometry.KDTreeFlann.search_knn_vector_3d
b.inputs.permute
self._integrate_func
self.get_volume
get_view_frustum
utils.SaveScene.save_scene_eval
tools.render.Visualizer
min_c.max_c.int

@developer Could please help me check this issue? May I pull a request to fix it? Thank you very much.

opened by PyDeps 0

Owner

ZJU3DV

ZJU3DV is a research group of State Key Lab of CAD&CG, Zhejiang University. We focus on the research of 3D computer vision, SLAM and AR.

GitHub https://zju3dv.github.io/neuralrecon/

Code for CVPR 2021 oral paper "Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contexts"

Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contexts The rapid progress in 3D scene understanding has come with growing dem

182 Dec 30, 2022

Code for "Reconstructing 3D Human Pose by Watching Humans in the Mirror", CVPR 2021 oral

Reconstructing 3D Human Pose by Watching Humans in the Mirror Qi Fang*, Qing Shuai*, Junting Dong, Hujun Bao, Xiaowei Zhou CVPR 2021 Oral The videos a

178 Dec 13, 2022

This is the code for CVPR 2021 oral paper: Jigsaw Clustering for Unsupervised Visual Representation Learning

JigsawClustering Jigsaw Clustering for Unsupervised Visual Representation Learning Pengguang Chen, Shu Liu, Jiaya Jia Introduction This project provid

73 Sep 18, 2022

Official pytorch implementation of paper "Inception Convolution with Efficient Dilation Search" (CVPR 2021 Oral).

IC-Conv This repository is an official implementation of the paper Inception Convolution with Efficient Dilation Search. Getting Started Download Imag

111 Dec 31, 2022

Official PyTorch implementation of RobustNet (CVPR 2021 Oral)

RobustNet (CVPR 2021 Oral): Official Project Webpage Codes and pretrained models will be released soon. This repository provides the official PyTorch

173 Dec 21, 2022

Dynamic Slimmable Network (CVPR 2021, Oral)

Dynamic Slimmable Network (DS-Net) This repository contains PyTorch code of our paper: Dynamic Slimmable Network (CVPR 2021 Oral). Architecture of DS-

197 Dec 9, 2022

[CVPR 2021 Oral] ForgeryNet: A Versatile Benchmark for Comprehensive Forgery Analysis

ForgeryNet: A Versatile Benchmark for Comprehensive Forgery Analysis ForgeryNet: A Versatile Benchmark for Comprehensive Forgery Analysis [arxiv|pdf|v

78 Dec 22, 2022

[CVPR 2021 Oral] Variational Relational Point Completion Network

VRCNet: Variational Relational Point Completion Network This repository contains the PyTorch implementation of the paper: Variational Relational Point

121 Dec 12, 2022

Quasi-Dense Similarity Learning for Multiple Object Tracking, CVPR 2021 (Oral)

Quasi-Dense Tracking This is the offical implementation of paper Quasi-Dense Similarity Learning for Multiple Object Tracking. We present a trailer th

327 Dec 27, 2022

Pytorch implementation for "Adversarial Robustness under Long-Tailed Distribution" (CVPR 2021 Oral)

Adversarial Long-Tail This repository contains the PyTorch implementation of the paper: Adversarial Robustness under Long-Tailed Distribution, CVPR 20

89 Dec 15, 2022

TAP: Text-Aware Pre-training for Text-VQA and Text-Caption, CVPR 2021 (Oral)

TAP: Text-Aware Pre-training TAP: Text-Aware Pre-training for Text-VQA and Text-Caption by Zhengyuan Yang, Yijuan Lu, Jianfeng Wang, Xi Yin, Dinei Flo

61 Nov 14, 2022

Official PyTorch Implementation of Convolutional Hough Matching Networks, CVPR 2021 (oral)

Convolutional Hough Matching Networks This is the implementation of the paper "Convolutional Hough Matching Network" by J. Min and M. Cho. Implemented

70 Nov 22, 2022

Implementation of "Bidirectional Projection Network for Cross Dimension Scene Understanding" CVPR 2021 (Oral)

Bidirectional Projection Network for Cross Dimension Scene Understanding CVPR 2021 (Oral) [ Project Webpage ] [ arXiv ] [ Video ] Existing segmentatio

135 Dec 26, 2022

Deep RGB-D Saliency Detection with Depth-Sensitive Attention and Automatic Multi-Modal Fusion (CVPR'2021, Oral)

DSA^2 F: Deep RGB-D Saliency Detection with Depth-Sensitive Attention and Automatic Multi-Modal Fusion (CVPR'2021, Oral) This repo is the official imp

46 Dec 21, 2022

🔥RandLA-Net in Tensorflow (CVPR 2020, Oral & IEEE TPAMI 2021)

RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds (CVPR 2020) This is the official implementation of RandLA-Net (CVPR2020, Oral

1k Dec 30, 2022

git git《Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking》(CVPR 2021) GitHub:git2] 《Masksembles for Uncertainty Estimation》(CVPR 2021) GitHub:git3]

Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking Ning Wang, Wengang Zhou, Jie Wang, and Houqiang Li Accepted by CVPR

236 Dec 22, 2022

Official code for "End-to-End Optimization of Scene Layout" -- including VAE, Diff Render, SPADE for colorization (CVPR 2020 Oral)

End-to-End Optimization of Scene Layout Code release for: End-to-End Optimization of Scene Layout CVPR 2020 (Oral) Project site, Bibtex For help conta

41 Dec 9, 2022

Code repo for realtime multi-person pose estimation in CVPR'17 (Oral)

Realtime Multi-Person Pose Estimation By Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh. Introduction Code repo for winning 2016 MSCOCO Keypoints Cha

4.9k Dec 31, 2022

Code for "PVNet: Pixel-wise Voting Network for 6DoF Pose Estimation" CVPR 2019 oral

Good news! We release a clean version of PVNet: clean-pvnet, including how to train the PVNet on the custom dataset. Use PVNet with a detector. The tr

722 Dec 27, 2022