ADOP: Approximate Differentiable One-Pixel Point Rendering

Related tags

Deep Learning ADOP
Overview

ADOP: Approximate Differentiable One-Pixel Point Rendering

Abstract: We present a novel point-based, differentiable neural rendering pipeline for scene refinement and novel view synthesis. The input are an initial estimate of the point cloud and the camera parameters. The output are synthesized images from arbitrary camera poses. The point cloud rendering is performed by a differentiable renderer using multi-resolution one-pixel point rasterization. Spatial gradients of the discrete rasterization are approximated by the novel concept of ghost geometry. After rendering, the neural image pyramid is passed through a deep neural network for shading calculations and hole-filling. A differentiable, physically-based tonemapper then converts the intermediate output to the target image. Since all stages of the pipeline are differentiable, we optimize all of the scene's parameters i.e. camera model, camera pose, point position, point color, environment map, rendering network weights, vignetting, camera response function, per image exposure, and per image white balance. We show that our system is able to synthesize sharper and more consistent novel views than existing approaches because the initial reconstruction is refined during training. The efficient one-pixel point rasterization allows us to use arbitrary camera models and display scenes with well over 100M points in real time.

  • The source code will be published after the paper has been accepted to a conference.

[Full Paper]

Video

Watch the video

Comments
  • Guide on exporting scene from COLMAP

    Guide on exporting scene from COLMAP

    Hi! I have some troubles exporting point cloud from COLMAP to compatible with ADOP format. Here is what I did:

    1. Autoreconstruction in COLMAP (both sparse and dense)
    2. File > Export As > as .ply
    3. preprocessed it with colmap2adop tool
    4. started training
    5. got an error that my point cloud file does not contain normal data
    6. I checked that your .ply files have normal data in headers, but I struggle to make COLMAP include it in mine
    opened by DenShlk 22
  • RAM usage when training dataset

    RAM usage when training dataset

    Through a lot of trial and error I managed to get both the ADOP viewer and trainer set up on Ubuntu 20.04.

    SETUP I'm trying to run my own dataset through the trainer to see how it is. I only have an Nvidia GTX 1070 card with 8GB of VRAM, 16GB of physical RAM but plenty of hard disk space. I've gotten to the point that the trainer runs without crashing. My dataset is 175 images, downscaled to 1000x562, converted from a ~7M point-cloud project in COLMAP.

    PROBLEM However, running the trainer seems to take up over 1GB per epoch on top of the base amount. This quickly sends my RAM usage to near full. Luckily I have a large swap partition but that too fills up over hundreds of epochs. Is this expected behaviour or a memory leak? Or could it be because my install or conda environment might be faulty? Or is it my incapable hardware? In any case, once all my swap space gets filled the program is unfortunately killed on its own.

    If tweaking any of the training config or dataset parameters could potentially fix or mitigate this issue, please do let me know. Thank you for your work on this, I find this to be a very interesting project.

    opened by ShinyLuxray 19
  • How to optimize for rendering

    How to optimize for rendering

    Hi I have a question regarding the adop_viewer. l tried the adop_viewer with pretrained models on tanks and temples dataset, it works perfectly on a laptop with a 3080 GPU (8GB VRAM) on all scenes.

    I guess you are using the 40GB A100 ini files for training these models, so I also tried training my data on a GPU with similar VRAM using exact settings from train_tank_and_temples_multi.ini. However, when I attempt to visualize my checkpoints on the 3080 GPU it shows CUDA out of memory for all my scenes.

    My question is: for rendering, did you make some optimizations to make the trained models fit in a GPU with less VRAM? Or is there certain configurations/arguments that allows the adop_viewer to take a model trained with high-end GPUs and render on low-end GPUs? Thanks in advance!

    opened by qiaosongwang 12
  • Issue with multi-camera COLMAP dataset

    Issue with multi-camera COLMAP dataset

    I have successfully trained with ADOP on scenes that use a single camera. I am having difficulty, however, with a larger dataset comprised of 882 cameras, 2/3 of which are from drone footage, the other 1/3 being a hand-held DSLR.

    COLMAP successfully generates the camera registrations and point cloud, and running colmap2adop correctly creates all the necessary files for training. I am noticing, however, that COLMAP resizes every single image, effectively generating 882 individual camera profiles. I thought at first this wouldn't necessarily be an issue, as adop_train can parse multiple cameras. I am running into this error, though:

    Assertion 'cam.w == image_size_input(0)' failed!
  File: /home/visionarymind/Downloads/ADOP/src/lib/data/Dataset.cpp:49
  Function: SceneDataTrainSampler::SceneDataTrainSampler(std::shared_ptr<SceneData>, std::vector<int>, bool, Saiga::ivec2, int, bool)
Aborted (core dumped)

    I have looked at each camera.ini file, and the dimensions match each image. Would you have any idea why this error would be thrown and any way to work around it? Technically, there were only two cameras, but COLMAP resizes the images during registration.

    opened by VisionaryMind 10
  • Something worry in PointRenderer.cu & PointRenderer.h

    Something worry in PointRenderer.cu & PointRenderer.h

    2021-11-10 15-59-34屏幕截图 Can not Build CUDA object src/lib/CMakeFiles/NeuralPoints.dir/rendering/PointRenderer.cu.o

    /home/xgenie/projectfold/ADOP/src/lib/./config.h(25): error: inline specifier allowed on function declarations only

    /home/xgenie/projectfold/ADOP/src/lib/./config.h(25): error: inline specifier allowed on function declarations only

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(37): error: qualified name is not allowed

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(39): error: Function is not a template

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(39): error: not a class or struct name

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(42): error: identifier "variable_list" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(42): error: identifier "AutogradContext" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(42): error: identifier "Variable" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(42): error: identifier "Variable" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(42): error: identifier "Variable" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(43): error: identifier "Variable" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(43): error: identifier "Variable" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(43): error: identifier "IValue" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(45): error: identifier "variable_list" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(45): error: identifier "AutogradContext" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(45): error: identifier "variable_list" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(422): error: expected an identifier

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(422): error: identifier "K" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(422): error: expected a "]"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(422): warning: nonstandard use of "auto" to both deduce the type from an initializer and to announce a trailing return type

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(422): error: cannot deduce "auto" type (initializer required)

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(422): error: expected a ";"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(429): error: expected an identifier

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(429): error: identifier "aff" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(429): error: expected a "]"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(429): warning: nonstandard use of "auto" to both deduce the type from an initializer and to announce a trailing return type

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(429): error: cannot deduce "auto" type (initializer required)

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(429): error: expected a ";"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(621): error: expected an identifier

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(621): error: identifier "K" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(621): error: expected a "]"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(621): warning: nonstandard use of "auto" to both deduce the type from an initializer and to announce a trailing return type

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(621): error: cannot deduce "auto" type (initializer required)

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(621): error: expected a ";"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(622): error: expected an identifier

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(622): error: identifier "g_point" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(622): error: expected a "]"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(622): warning: nonstandard use of "auto" to both deduce the type from an initializer and to announce a trailing return type

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(622): error: cannot deduce "auto" type (initializer required)

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(622): error: expected a ";"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(686): error: expected an identifier

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(686): error: identifier "aff" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(686): error: expected a "]"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(686): warning: nonstandard use of "auto" to both deduce the type from an initializer and to announce a trailing return type

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(686): error: cannot deduce "auto" type (initializer required)

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(686): error: expected a ";"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(687): error: expected an identifier

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(687): error: identifier "g_point" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(687): error: expected a "]"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(687): warning: nonstandard use of "auto" to both deduce the type from an initializer and to announce a trailing return type

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(687): error: cannot deduce "auto" type (initializer required)

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(687): error: expected a ";"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(527): warning: variable "iw" was declared but never referenced

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(615): warning: variable "cam2" was set but never used

    opened by XGenietony 10
  • Unable to get output in the neural view

    Unable to get output in the neural view

    Hello, thanks for releasing your code!

    I tried running the program (the compilation was successful) but I am unable to obtain any output in the neural view.

    image

    This is the output I get from the terminal. I am using Ubuntu 20.04.3 LTS.

    (base) jeremy@jeremy-ubuntu:~/Documents/ADOP$ ./build/bin/adop_viewer --scene_dir scenes/tt_lighthouse/
register neural render info
Ref. 
=============== Saiga ===============
| Saiga Version     1.3.2           |
| Eigen Version     3.3.91          |
| Compiler          GNU             |
|   -> Version      9.3.0           |
| Build Type        RelWithDebInfo  |
| Debug             0               |
| Eigen Debug       0               |
| ASAN              0               |
| Asserts           1               |
| Optimizations     0               |
=====================================
Initializing GLFW.
Initializing GLFW sucessfull!
Creating GLFW Window. 1920x1080 Mode=3 Fullscreen=1 Borderless=0
=========================== OpenGL ===========================
| OpenGL Version    3.2.0 NVIDIA 495.29.05                   | 
| GLSL Version      1.50 NVIDIA via Cg compiler              | 
| Renderer          NVIDIA GeForce GTX 1660 SUPER/PCIe/SSE2  | 
| Vendor            NVIDIA Corporation                       | 
==============================================================
[Renderer] Target resized to 1848x1016
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/imgui_gl.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/tone_map.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/tone_map_linear.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/bloom_extract_bright.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/bloom_downsample.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/bloom_upsample.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/bloom_combine_simple.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/copy_image.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/compute_blur.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/compute_blur.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/light_directional.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/light_directional.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/light_point.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/light_point.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/light_spot.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/light_spot.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/light_point.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/light_spot.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/stenciltest.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/post_processing/post_processing.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/blitDepth.glsl
Deferred Renderer initialized. Render resolution: 1848x1016
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/post_processing/imagedisplay.glsl
Program Initialized!
Loading Scene scenes/tt_lighthouse/
====================================
Scene Loaded
  Name       tt_lighthouse
  Path       /home/jeremy/Documents/ADOP/scenes/tt_lighthouse
  Image Size 2048x1080
  Aspect     1.8963
  K          1666.68 1661.18 1024    540     0      
  ocam       2048x1080 affine(1, 0, 0, 0, 0) cam2world() world2cam()
  ocam cut   1
  normalized center 0 0
  dist       -0.126731   0.0217503   0           0           0           0           0.000406465 0.000123456
  Points     12313620
  Colors     1
  Normals    1
  Num Images 309
  Num Cameras 1
====================================
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/asset/ColoredAsset.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/asset/ColoredAsset.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/asset/ColoredAsset.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/asset/LineVertexColoredAsset.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/asset/LineVertexColoredAsset.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/asset/LineVertexColoredAsset.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/tone_map.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/tone_map_linear.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/bloom_extract_bright.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/bloom_downsample.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/bloom_upsample.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/bloom_combine_simple.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/copy_image.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/compute_blur.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/compute_blur.glsl
Found experiment experiments//2021-10-30_09:17:38_multi_scene_pose/ with 1 epochs
Found experiment experiments//2021-10-30_09:14:22_boat/ with 1 epochs
loading checkpoint 2021-10-30_09:17:38_multi_scene_pose -> ep0400
Load Checkpoint render
Total Model Params: 574651
> Reset cuda cache
GPU memory - Point Cloud: 295.527MB
Pinhole Intrinsics:
Tensor [1, 13] float cuda:0 Min/Max -0.126731 1666.68 Mean 376.289 Sum 4891.75 req-grad 1
GPU memory - Texture: 197.018MB
Load Checkpoint points 12313620 max uv: 12313619
Load Checkpoint texture. Texels: 12313620 Channels: 4
Load Checkpoint pose
First pose before SE3(Quatwxyz(0.990142,-0.0887136,0.107975,0.00948205),Vec3(0.115149,0.198961,4.53537))
First pose after SE3(Quatwxyz(0.990146,-0.0887041,0.107948,0.00948975),Vec3(0.114891,0.198873,4.53539))
Load Checkpoint intrinsics
Load Checkpoint environment_map
Load Checkpoint vignette
Load Checkpoint response
Load Checkpoint exposures_values
Using Adam texture optimzier
optimizing texture with lr 0.08/0.004
optimizing environment_map with lr 0.02
optimizing response with lr 0.001
optimizing exposure with lr 0.0005
optimizing vignette with lr 5e-06
optimizing poses with lr 0.005
GPU memory - Texture: 197.018MB
Current Best (img,cam) = (0,0) EV: -0.239136
[Renderer] Target resized to 733x354
[DeferredRenderer] Resize 733x354
loading shader shader//point_render.glsl
[Renderer] Target resized to 545x249
[DeferredRenderer] Resize 545x249
loading checkpoint 2021-10-30_09:17:38_multi_scene_pose -> ep0400
Load Checkpoint render
Total Model Params: 574651
> Reset cuda cache
GPU memory - Point Cloud: 295.527MB
Pinhole Intrinsics:
Tensor [1, 13] float cuda:0 Min/Max -0.126731 1666.68 Mean 376.289 Sum 4891.75 req-grad 1
GPU memory - Texture: 197.018MB
Load Checkpoint points 12313620 max uv: 12313619
Load Checkpoint texture. Texels: 12313620 Channels: 4
Load Checkpoint pose
First pose before SE3(Quatwxyz(0.990146,-0.0887041,0.107948,0.00948975),Vec3(0.114891,0.198873,4.53539))
First pose after SE3(Quatwxyz(0.990146,-0.0887041,0.107948,0.00948975),Vec3(0.114891,0.198873,4.53539))
Load Checkpoint intrinsics
Load Checkpoint environment_map
Load Checkpoint vignette
Load Checkpoint response
Load Checkpoint exposures_values
Using Adam texture optimzier
optimizing texture with lr 0.08/0.004
optimizing environment_map with lr 0.02
optimizing response with lr 0.001
optimizing exposure with lr 0.0005
optimizing vignette with lr 5e-06
optimizing poses with lr 0.005
GPU memory - Texture: 197.018MB
Current Best (img,cam) = (0,0) EV: -0.239136

    Any help would be greatly appreciated! Thank you!

    opened by mickey1356 9
  • A question about CmakeList.txt

    A question about CmakeList.txt

    Hello, I am a beginner of neural network. I want to run your program on my computer, but when I Compiling ADOP,in the command "cmake -DCMAKE_PREFIX_PATH="${CONDA}/lib/python3.8/site-packages/torch/;${CONDA}" ..",there was an error,it was "The source directory XXXXdoes not appear to contain CMakeLists.txt."How can i solve this problem?Should i write the CmakeList on myself ?Is there sth wrong in other parts ?this may easy for most people,but i sincerely hope you can answer me,thank you!

    opened by Lu-pin-an 9
  • Getting

    Getting "Loss not finite :(".

    Hello, I'm preparing a dataset of the interior of an appartment. I checked and COLMAP seems to work well, despite that there is a fair amount of white walls. After a random number of epochs, I'm getting the following (pasting only the tail of the output):

    === Epoch 81 ===

Train 81 |   0% |                              |    0/1016 [00:00:0000] [0.00 e/s] 
Train 81 |   0% |                              |    0/1016 [00:00:0000] [0.00 e/s] 
Train 81 |   4% |#                             |   40/1016 [00:05:0000] [8.00 e/s]  Cur=212.233490 Avg=302.391327
Train 81 |   9% |##                            |   88/1016 [00:10:0000] [8.80 e/s]  Cur=333.448730 Avg=287.600220
Train 81 |  13% |####                          |  136/1016 [00:15:0000] [9.07 e/s]  Cur=441.840668 Avg=295.512512
Train 81 |  18% |#####                         |  184/1016 [00:20:0000] [9.20 e/s]  Cur=379.657135 Avg=317.960571
Train 81 |  22% |######                        |  224/1016 [00:25:0000] [8.96 e/s]  Cur=277.145172 Avg=296.512878
Train 81 |  27% |########                      |  272/1016 [00:30:0000] [9.07 e/s]  Cur=310.707367 Avg=302.186951
Train 81 |  31% |#########                     |  320/1016 [00:35:0000] [9.14 e/s]  Cur=34.233173 Avg=287.412201
Train 81 |  36% |##########                    |  368/1016 [00:40:0001] [9.20 e/s]  Cur=33.885960 Avg=282.051544
Train 81 |  41% |############                  |  416/1016 [00:45:0001] [9.24 e/s]  Cur=267.568176 Avg=272.566742
Train 81 |  46% |#############                 |  464/1016 [00:50:0001] [9.28 e/s]  Cur=376.509857 Avg=253.497742
Train 81 |  50% |###############               |  512/1016 [00:55:0001] [9.31 e/s]  Cur=347.692017 Avg=255.870377
Train 81 |  55% |################              |  560/1016 [01:00:0001] [9.33 e/s]  Cur=222.076523 Avg=251.570724
Train 81 |  60% |#################             |  608/1016 [01:05:0001] [9.35 e/s]  Cur=184.181763 Avg=259.685944
Train 81 |  65% |###################           |  656/1016 [01:10:0002] [9.37 e/s]  Cur=30.161480 Avg=255.755127
Train 81 |  69% |####################          |  704/1016 [01:15:0002] [9.39 e/s]  Cur=34.554646 Avg=257.362579
Train 81 |  74% |######################        |  752/1016 [01:20:0002] [9.40 e/s]  Cur=239.453049 Avg=248.403091
Train 81 |  78% |#######################       |  792/1016 [01:25:0002] [9.32 e/s]  Cur=33.262161 Avg=248.473404
Train 81 |  83% |########################      |  840/1016 [01:30:0002] [9.33 e/s]  Cur=348.131836 Avg=255.551346
Train 81 |  87% |##########################    |  888/1016 [01:35:0003] [9.35 e/s]  Cur=547.986633 Avg=261.241058
Train 81 |  92% |###########################   |  936/1016 [01:40:0003] [9.36 e/s]  Cur=268.251221 Avg=261.505127
Tensor [8, 4, 512, 512] float cuda:0 Min/Max 1.3125e-06 20599.7 Mean 0.520595 Sum 4.36707e+06 sdev 7.16297 req-grad 1
Tensor [8, 4, 256, 256] float cuda:0 Min/Max 3.05836e-05 34.5411 Mean 0.535766 Sum 1.12358e+06 sdev 0.771836 req-grad 1
Tensor [8, 4, 128, 128] float cuda:0 Min/Max 1.22896e-05 33.5968 Mean 0.552693 Sum 289770 sdev 0.658228 req-grad 1
Tensor [8, 4, 64, 64] float cuda:0 Min/Max 0.000194362 36.972 Mean 0.557636 Sum 73090.4 sdev 0.61885 req-grad 1

Tensor [8, 2, 512, 512] float cuda:0 Min/Max -1 0.999414 Mean 0.083806 Sum 351508 sdev 0.473772 req-grad 0
Tensor [1] float cuda:0 Min/Max nan nan Mean nan Sum nan sdev nan req-grad 1
Tensor [8, 3, 512, 512] float cuda:0 Min/Max nan nan Mean nan Sum nan sdev nan req-grad 1
Tensor [1, 1, 512, 512] float cuda:0 Min/Max 0 1 Mean 0.878906 Sum 230400 sdev 0.326237 req-grad 0
Tensor [8, 3, 512, 512] float cuda:0 Min/Max 0 0.956863 Mean 0.653581 Sum 4.11197e+06 sdev 0.274293 req-grad 0

Scene:
Scene Log - Texture: Tensor [4, 11655018] float cuda:0 Min/Max -22.6205 15.4725 Mean -0.8064 Sum -3.75944e+07 sdev 1.10892 req-grad 1
Background Desc:  -0.970885 -0.863956 -0.900406 -0.882389 
Environment map: Tensor [1, 4, 1024, 512] float cuda:0 Min/Max -6.2707 3.55126 Mean -0.982767 Sum -2.06101e+06 sdev 0.274843 req-grad 1
Poses: Tensor [254, 8] double cuda:0 Min/Max -6.83462 8.02766 Mean 0.196939 Sum 400.181 sdev 1.62054 req-grad 0
Vignette params:  -0.0149647 -0.00850714 -0.00492687 |   0.00124693 -0.000540384
terminate called after throwing an instance of 'std::runtime_error'
  what():  Loss not finite :(

    Could it be due to the lack of enough points on the white walls? If so, how would you go about it? I'm willing to provide any data if required to better assess this.

    opened by cduguet 8
  • Something wrong with the neural view using my own data

    Something wrong with the neural view using my own data

    After using the ADOP viewer with 2 kinds of my trained data, I got the following results.

    2022-06-29 10-50-44 的屏幕截图

    2022-06-29 11-08-36 的屏幕截图

    All neural views of them are definitely wrong. Is there the same problem with me?

    By the way, when I used the ADOP viewer with the scene of "boat", there is a correct colored display in the neural view.

    opened by HandsomeFYM 5
  • Strange random color when training

    Strange random color when training

    Hi，I have some doubt of my train result on other dataset. It looks like below:( left picture is the train result, right is ground trurh) image

    I can not find out why it happens, is it caused by the rasterization method?

    Below show some of the rasterization result:( left: train render result, medium: rgb ground truth, right: rasterization mask) image

    opened by Sylvia6 4
  • How to generate a video from a given trajectory?

    How to generate a video from a given trajectory?

    Hi, I saw you provided some videos for the trained models on the different scenes. I was wondering how you generated them? Is it possible to generate them with the viewer?

    opened by vahidEttehadi 4
  • How to use the mask function？

    How to use the mask function？

    Hi, I'm wondering to use the mask to filter out the dynamic objects in my scene. But how to use the mask function？I have set the mask_dir in the dataset.ini? Should I also specify the classifaction_dir in the dataset.ini？

    opened by Gatsby23 0
  • can't move in vr_viewer using oculus quest 2

    can't move in vr_viewer using oculus quest 2

    Thank you for the excellent work. VR is being tested.

    The build is successful, but moving the VR headset does not change the rendering screen.

    If there is a solution, please share it with me.

    opened by minsoo5403 1
  • fail to run ./build/bin/adop_viewer --scene_dir scenes/boat

    fail to run ./build/bin/adop_viewer --scene_dir scenes/boat

    optimizing texture with lr 0.01/0.0002 optimizing environment_map with lr 0.002 optimizing response with lr 0.001 optimizing vignette with lr 5e-06 optimizing poses with lr 0.005 signal 11 (Segmentation fault), address is 0x31f from 0x7fbbb8c33eb5 [bt]: (0) /host/home//G/1workspace/NeRF-space/ADOP/build/External/saiga/lib/libsaiga_core.so(_ZN5Saiga17printCurrentStackEv+0x38) [0x7fbbb7c3b468] [bt]: (1) /host/home//G/1workspace/NeRF-space/ADOP/build/External/saiga/lib/libsaiga_core.so(+0xe5871) [0x7fbbb7c3b871] [bt]: (2) /usr/lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7fbba6312420] [bt]: (3) /lib64/ld-linux-x86-64.so.2(+0x17eb5) [0x7fbbb8c33eb5] [bt]: (4) /usr/lib/x86_64-linux-gnu/libc.so.6(_dl_catch_exception+0x88) [0x7fbb96bf0928] [bt]: (5) /usr/lib/x86_64-linux-gnu/libc.so.6(_dl_catch_error+0x33) [0x7fbb96bf09f3] [bt]: (6) /usr/lib/x86_64-linux-gnu/libdl.so.2(+0x1b59) [0x7fbb96947b59] [bt]: (7) /usr/lib/x86_64-linux-gnu/libdl.so.2(dlclose+0x28) [0x7fbb96947468] [bt]: (8) /lib/x86_64-linux-gnu/libcuda.so(+0x217ec1) [0x7fbb31383ec1] [bt]: (9) /lib/x86_64-linux-gnu/libcuda.so(+0x217f33) [0x7fbb31383f33] [bt]: (10) /lib/x86_64-linux-gnu/libcuda.so(+0x2f42a2) [0x7fbb314602a2] [bt]: (11) /opt/conda/envs/adop/lib/libcudart.so.11.0(+0x1b408) [0x7fbbb6553408] [bt]: (12) /opt/conda/envs/adop/lib/libcudart.so.11.0(cudaGraphicsGLRegisterImage+0x1d1) [0x7fbbb65a34c1] [bt]: (13) /host/home//G/1workspace/NeRF-space/ADOP/build/bin/libNeuralPoints.so(_ZN5Saiga4CUDA7Interop9initImageEjN2gl6GLenumE+0x32) [0x7fbbb8b126b2] [bt]: (14) /host/home//G/1workspace/NeRF-space/ADOP/build/bin/libNeuralPoints.so(_ZN16RealTimeRenderer6RenderE9ImageInfo+0x78b) [0x7fbbb8affd2b] [bt]: (15) /host/home//G/1workspace/NeRF-space/ADOP/build/bin/libNeuralPoints.so(_ZN16RealTimeRenderer7ForwardEPN5Saiga6CameraE9ImageInfo+0x32b) [0x7fbbb8b03f5b] [bt]: (16) ./build/bin/adop_viewer(+0x21e07) [0x55e456e21e07] [bt]: (17) /host/home//G/1workspace/NeRF-space/ADOP/build/External/saiga/lib/libsaiga_opengl.so(_ZN5Saiga14OpenGLRenderer12PrepareImguiEb+0x160) [0x7fbbb87e7510] [bt]: (18) /host/home//G/1workspace/NeRF-space/ADOP/build/External/saiga/lib/libsaiga_opengl.so(_ZN5Saiga14OpenGLRenderer6renderERKNS_10RenderInfoE+0x5b) [0x7fbbb87e834b] [bt]: (19) /host/home//G/1workspace/NeRF-space/ADOP/build/External/saiga/lib/libsaiga_core.so(_ZN5Saiga10WindowBase6renderEv+0x4c) [0x7fbbb7c1745c] [bt]: (20) /host/home//G/1workspace/NeRF-space/ADOP/build/External/saiga/lib/libsaiga_core.so(_ZN5Saiga8MainLoop6renderEff+0x7a) [0x7fbbb7c1561a] [bt]: (21) /host/home//G/1workspace/NeRF-space/ADOP/build/External/saiga/lib/libsaiga_core.so(_ZN5Saiga8MainLoop13startMainLoopENS_18MainLoopParametersE+0x25c) [0x7fbbb7c159fc] [bt]: (22) /host/home//G/1workspace/NeRF-space/ADOP/build/External/saiga/lib/libsaiga_core.so(_ZN5Saiga10WindowBase13startMainLoopENS_18MainLoopParametersE+0x2d) [0x7fbbb7c1788d] [bt]: (23) ./build/bin/adop_viewer(+0x1adfa) [0x55e456e1adfa] [bt]: (24) /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fbb96ab4083] [bt]: (25) ./build/bin/adop_viewer(+0x1b1de) [0x55e456e1b1de] Assertion '0' failed! File: /host/home//G/1workspace/NeRF-space/ADOP/External/saiga/src/saiga/core/util/crash.cpp:109 Function: void Saiga::crit_err_hdlr(int, siginfo_t*, void*) Aborted (core dumped)

    opened by hongdayu 1
  • python setup.py install gives us an error

    python setup.py install gives us an error

    Building wheel torch-1.10.0a0+git302ee7b Traceback (most recent call last): File "setup.py", line 309, in cmake = CMake() File "/root/autodl-tmp/ADOP/External/pytorch/tools/setup_helpers/cmake.py", line 102, in init self._cmake_command = CMake._get_cmake_command() File "/root/autodl-tmp/ADOP/External/pytorch/tools/setup_helpers/cmake.py", line 126, in _get_cmake_command elif cmake is not None and CMake._get_version(cmake) >= distutils.version.LooseVersion("3.10.0"): File "/root/autodl-tmp/ADOP/External/pytorch/tools/setup_helpers/cmake.py", line 137, in _get_version return distutils.version.LooseVersion(line.strip().split(' ')[2]) AttributeError: module 'distutils' has no attribute 'version'

    opened by CHenYYff 0
  • make -j10 Failed

    make -j10 Failed

    image Thanks for your excellent work! But when I compile the adop project using "make -j10", it seems like meet errors. I'm following the steps of "compile instructions". Please help me out! Thanks~

    opened by Yukun66 0
Releases(v1.0)
Owner
Darius Rückert
Darius Rückert
GitHub
This repository contains the code for the CVPR 2020 paper "Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision"

Differentiable Volumetric Rendering Paper | Supplementary | Spotlight Video | Blog Entry | Presentation | Interactive Slides | Project Page This repos

null 697 Jan 6, 2023
Official Repo for ICCV2021 Paper: Learning to Regress Bodies from Images using Differentiable Semantic Rendering

[ICCV2021] Learning to Regress Bodies from Images using Differentiable Semantic Rendering Getting Started DSR has been implemented and tested on Ubunt

Sai Kumar Dwivedi 83 Nov 27, 2022
Differentiable Neural Computers, Sparse Access Memory and Sparse Differentiable Neural Computers, for Pytorch

Differentiable Neural Computers and family, for Pytorch Includes: Differentiable Neural Computers (DNC) Sparse Access Memory (SAM) Sparse Differentiab

ixaxaar 302 Dec 14, 2022
Rendering Point Clouds with Compute Shaders

Compute Shader Based Point Cloud Rendering This repository contains the source code to our techreport: Rendering Point Clouds with Compute Shaders and

Markus Schütz 460 Jan 5, 2023
pyhsmm - library for approximate unsupervised inference in Bayesian Hidden Markov Models (HMMs) and explicit-duration Hidden semi-Markov Models (HSMMs), focusing on the Bayesian Nonparametric extensions, the HDP-HMM and HDP-HSMM, mostly with weak-limit approximations.

Bayesian inference in HSMMs and HMMs This is a Python library for approximate unsupervised inference in Bayesian Hidden Markov Models (HMMs) and expli

Matthew Johnson 527 Dec 4, 2022
Approximate Nearest Neighbors in C++/Python optimized for memory usage and loading/saving to disk

Annoy Annoy (Approximate Nearest Neighbors Oh Yeah) is a C++ library with Python bindings to search for points in space that are close to a given quer

Spotify 10.6k Jan 4, 2023
TorchPQ is a python library for Approximate Nearest Neighbor Search (ANNS) and Maximum Inner Product Search (MIPS) on GPU using Product Quantization (PQ) algorithm.

Efficient implementations of Product Quantization and its variants using Pytorch and CUDA

null 146 Dec 28, 2022
Optimal space decomposition based-product quantization for approximate nearest neighbor search

Optimal space decomposition based-product quantization for approximate nearest neighbor search Abstract Product quantization(PQ) is an effective neare

Mylove 1 Nov 19, 2021
Fastshap: A fast, approximate shap kernel

fastshap: A fast, approximate shap kernel fastshap was designed to be: Fast Calculating shap values can take an extremely long time. fastshap utilizes

Samuel Wilson 22 Sep 24, 2022
Convert Table data to approximate values with GUI

Table_Editor Convert Table data to approximate values with GUIs... usage - Import methods for extension Tables. Imported method supposed to have only

CLJ 1 Jan 10, 2022
Personal implementation of paper "Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval"

Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval This repo provides personal implementation of paper Approximate Ne

John 8 Oct 7, 2022
Implementation of the "PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences" paper.

PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences Introduction Point cloud sequences are irregular and unordered in the spatial dimen

Hehe Fan 63 Dec 9, 2022
Code for "PV-RAFT: Point-Voxel Correlation Fields for Scene Flow Estimation of Point Clouds", CVPR 2021

PV-RAFT This repository contains the PyTorch implementation for paper "PV-RAFT: Point-Voxel Correlation Fields for Scene Flow Estimation of Point Clou

Yi Wei 43 Dec 5, 2022
Implementation of the "Point 4D Transformer Networks for Spatio-Temporal Modeling in Point Cloud Videos" paper.

Point 4D Transformer Networks for Spatio-Temporal Modeling in Point Cloud Videos Introduction Point cloud videos exhibit irregularities and lack of or

Hehe Fan 101 Dec 29, 2022
Point Cloud Denoising input segmentation output raw point-cloud valid/clear fog rain de-noised Abstract Lidar sensors are frequently used in environme

Point Cloud Denoising input segmentation output raw point-cloud valid/clear fog rain de-noised Abstract Lidar sensors are frequently used in environme

null 75 Nov 24, 2022
Synthetic LiDAR sequential point cloud dataset with point-wise annotations

SynLiDAR dataset: Learning From Synthetic LiDAR Sequential Point Cloud This is official repository of the SynLiDAR dataset. For technical details, ple

null 78 Dec 27, 2022
[ICCV 2021 Oral] SnowflakeNet: Point Cloud Completion by Snowflake Point Deconvolution with Skip-Transformer

This repository contains the source code for the paper SnowflakeNet: Point Cloud Completion by Snowflake Point Deconvolution with Skip-Transformer (ICCV 2021 Oral). The project page is here.

AllenXiang 65 Dec 26, 2022
Unofficial implementation of Point-Unet: A Context-Aware Point-Based Neural Network for Volumetric Segmentation

Point-Unet This is an unofficial implementation of the MICCAI 2021 paper Point-Unet: A Context-Aware Point-Based Neural Network for Volumetric Segment

Namt0d 9 Dec 7, 2022
Point-NeRF: Point-based Neural Radiance Fields

Point-NeRF: Point-based Neural Radiance Fields Project Sites | Paper | Primary c

Qiangeng Xu 662 Jan 1, 2023