nvdiffmodeling

Differentiable rasterization applied to 3D model simplification tasks, as described in the paper:

Appearance-Driven Automatic 3D Model Simplification
Jon Hasselgren, Jacob Munkberg, Jaakko Lehtinen, Miika Aittala and Samuli Laine
https://research.nvidia.com/publication/2021-04_Appearance-Driven-Automatic-3D
https://arxiv.org/abs/2104.03989

License

This work is made available under the Nvidia Source Code License.

For business inquiries, please contact [email protected]

Installation

Requires VS2019+, Cuda 10.2+ and PyTorch 1.6+

Tested in Anaconda3 with Python 3.6 and PyTorch 1.8

One time setup (Windows)

Install the Cuda toolkit (required to build the PyTorch extensions). We support Cuda 10.2 and above. Pick the appropriate version of PyTorch compatible with the installed Cuda toolkit. Below is an example with Cuda 11.1

conda create -n dmodel python=3.6
activate dmodel
conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c conda-forge
conda install imageio
pip install PyOpenGL glfw

Install nvdiffrast: https://github.com/NVlabs/nvdiffrast in the dmodel conda env.

Every new command prompt

activate dmodel

Examples

Sphere to cow example:

python train.py --config configs/spot.json

The results will be stored in the out folder. The Spot model was created and released into the public domain by Keenan Crane.

Additional assets can be downloaded here [205MB]. Unzip and place the subfolders in the project data folder, e.g., data\skull. All assets are copyright of their respective authors, see included license files for further details.

Included examples

skull.json - Joint normal map and shape optimization on a skull
ewer.json - Ewer model from a reduced mesh as initial guess
gardenina.json - Aggregate geometry example
hibiscus.json - Aggregate geometry example
figure_brushed_gold_64.json - LOD example, trained against a supersampled reference
figure_displacement.json - Joint shape, normal map, and displacement map example

The json files that end in _paper.json are configs with the settings used for the results in the paper. They take longer and require a GPU with sufficient memory.

Server usage (through Docker)

Build docker image (run the command from the code root folder). docker build -f docker/Dockerfile -t diffmod:v1 . Requires a driver that supports Cuda 10.1 or newer.
Start an interactive docker container: docker run --gpus device=0 -it --rm -v /raid:/raid -it diffmod:v1 bash
Detached docker: docker run --gpus device=1 -d -v /raid:/raid -w=[path to the code] diffmod:v1 python train.py --config configs/spot.json

I am trying to get this to work to test the workflow but ran into the error I have installed "Cuda" 11.1 and everything else as in the tutorial Graphics accelerator 3080t

After trying to run a test command, it gives me the following text Do you have any ideas how to fix this? I am more of a technical artist and a cursory search did not return much


(dmodel) I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main>python train.py --config configs/spot.json
Using C:\Users\alexp\AppData\Local\torch_extensions\torch_extensions\Cache as PyTorch extensions root...
C:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py:305: UserWarning: Error checking compiler version for cl: 'utf-8' codec can't decode byte 0x8e in position 0: invalid start byte
  warnings.warn(f'Error checking compiler version for {compiler}: {error}')
Detected CUDA files, patching ldflags
Emitting ninja build file C:\Users\alexp\AppData\Local\torch_extensions\torch_extensions\Cache\renderutils_plugin\build.ninja...
Building extension module renderutils_plugin...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/7] cl /showIncludes -DTORCH_EXTENSION_NAME=renderutils_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\TH -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\ProgramData\Anaconda3\envs\dmodel\Include -D_GLIBCXX_USE_CXX11_ABI=0 /MD /wd4819 /wd4251 /wd4244 /wd4267 /wd4275 /wd4018 /wd4190 /EHsc -c I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\c_src\common.cpp /Focommon.o
FAILED: common.o
cl /showIncludes -DTORCH_EXTENSION_NAME=renderutils_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\TH -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\ProgramData\Anaconda3\envs\dmodel\Include -D_GLIBCXX_USE_CXX11_ABI=0 /MD /wd4819 /wd4251 /wd4244 /wd4267 /wd4275 /wd4018 /wd4190 /EHsc -c I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\c_src\common.cpp /Focommon.o
Оптимизирующий компилятор Microsoft (R) C/C++ версии 19.25.28611 для x64
(C) Корпорация Майкрософт (Microsoft Corporation).  Все права защищены.

Примечание: включение файла:  C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include\cuda_runtime.h
Примечание: включение файла:   C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include\crt/host_config.h
Примечание: включение файла:   C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include\builtin_types.h
Примечание: включение файла:    C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include\device_types.h
Примечание: включение файла:     C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include\crt/host_defines.h
Примечание: включение файла:    C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include\driver_types.h
Примечание: включение файла:     C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include\crt/host_defines.h
Примечание: включение файла:     C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include\vector_types.h
Примечание: включение файла:      C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include\crt/host_defines.h
Примечание: включение файла:     C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\limits.h
Примечание: включение файла:      C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\vcruntime.h
Примечание: включение файла:       C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\sal.h
Примечание: включение файла:        C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\concurrencysal.h
Примечание: включение файла:       C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\vadefs.h
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include\driver_types.h(78): fatal error C1083: Не удается открыть файл включение: stddef.h: No such file or directory,
[2/7] cl /showIncludes -DTORCH_EXTENSION_NAME=renderutils_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\TH -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\ProgramData\Anaconda3\envs\dmodel\Include -D_GLIBCXX_USE_CXX11_ABI=0 /MD /wd4819 /wd4251 /wd4244 /wd4267 /wd4275 /wd4018 /wd4190 /EHsc -c I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\c_src\torch_bindings.cpp /Fotorch_bindings.o
FAILED: torch_bindings.o
cl /showIncludes -DTORCH_EXTENSION_NAME=renderutils_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\TH -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\ProgramData\Anaconda3\envs\dmodel\Include -D_GLIBCXX_USE_CXX11_ABI=0 /MD /wd4819 /wd4251 /wd4244 /wd4267 /wd4275 /wd4018 /wd4190 /EHsc -c I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\c_src\torch_bindings.cpp /Fotorch_bindings.o
Оптимизирующий компилятор Microsoft (R) C/C++ версии 19.25.28611 для x64
(C) Корпорация Майкрософт (Microsoft Corporation).  Все права защищены.

Примечание: включение файла:  C:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch/extension.h
Примечание: включение файла:   C:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include\torch/all.h
Примечание: включение файла:    C:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include\torch/cuda.h
Примечание: включение файла:     C:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch/csrc/WindowsTorchApiMacro.h
Примечание: включение файла:      C:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\c10/macros/Export.h
Примечание: включение файла:       C:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\c10/macros/cmake_macros.h
Примечание: включение файла:     C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\cstdint
Примечание: включение файла:      C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\yvals.h
Примечание: включение файла:       C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\yvals_core.h
Примечание: включение файла:        C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\vcruntime.h
Примечание: включение файла:         C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\sal.h
Примечание: включение файла:          C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\concurrencysal.h
Примечание: включение файла:         C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\vadefs.h
Примечание: включение файла:        C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\xkeycheck.h
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\yvals.h(12): fatal error C1083: Не удается открыть файл включение: crtdbg.h: No such file or directory,
[3/7] C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\bin\nvcc --generate-dependencies-with-compile --dependency-output mesh.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=renderutils_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\TH -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\ProgramData\Anaconda3\envs\dmodel\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -c I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\c_src\mesh.cu -o mesh.cuda.o
FAILED: mesh.cuda.o
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\bin\nvcc --generate-dependencies-with-compile --dependency-output mesh.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=renderutils_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\TH -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\ProgramData\Anaconda3\envs\dmodel\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -c I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\c_src\mesh.cu -o mesh.cuda.o
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\crtdefs.h(10): fatal error C1083: ═х єфрхЄё  юЄъЁ√Є№ Їрщы тъы■ўхэшх: corecrt.h: No such file or directory,
mesh.cu
[4/7] C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\bin\nvcc --generate-dependencies-with-compile --dependency-output loss.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=renderutils_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\TH -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\ProgramData\Anaconda3\envs\dmodel\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -c I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\c_src\loss.cu -o loss.cuda.o
FAILED: loss.cuda.o
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\bin\nvcc --generate-dependencies-with-compile --dependency-output loss.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=renderutils_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\TH -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\ProgramData\Anaconda3\envs\dmodel\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -c I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\c_src\loss.cu -o loss.cuda.o
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\crtdefs.h(10): fatal error C1083: ═х єфрхЄё  юЄъЁ√Є№ Їрщы тъы■ўхэшх: corecrt.h: No such file or directory,
loss.cu
[5/7] C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\bin\nvcc --generate-dependencies-with-compile --dependency-output bsdf.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=renderutils_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\TH -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\ProgramData\Anaconda3\envs\dmodel\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -c I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\c_src\bsdf.cu -o bsdf.cuda.o
FAILED: bsdf.cuda.o
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\bin\nvcc --generate-dependencies-with-compile --dependency-output bsdf.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=renderutils_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\TH -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\ProgramData\Anaconda3\envs\dmodel\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -c I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\c_src\bsdf.cu -o bsdf.cuda.o
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\crtdefs.h(10): fatal error C1083: ═х єфрхЄё  юЄъЁ√Є№ Їрщы тъы■ўхэшх: corecrt.h: No such file or directory,
bsdf.cu
[6/7] C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\bin\nvcc --generate-dependencies-with-compile --dependency-output normal.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=renderutils_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\TH -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\ProgramData\Anaconda3\envs\dmodel\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -c I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\c_src\normal.cu -o normal.cuda.o
FAILED: normal.cuda.o
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\bin\nvcc --generate-dependencies-with-compile --dependency-output normal.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=renderutils_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\TH -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\ProgramData\Anaconda3\envs\dmodel\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -c I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\c_src\normal.cu -o normal.cuda.o
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\crtdefs.h(10): fatal error C1083: ═х єфрхЄё  юЄъЁ√Є№ Їрщы тъы■ўхэшх: corecrt.h: No such file or directory,
normal.cu
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 1672, in _run_ninja_build
    env=env)
  File "C:\ProgramData\Anaconda3\envs\dmodel\lib\subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "train.py", line 20, in <module>
    import src.renderutils as ru
  File "I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\__init__.py", line 9, in <module>
    from .ops import xfm_points, xfm_vectors, image_loss, prepare_shading_normal, lambert, pbr_specular, pbr_bsdf, _fresnel_shlick, _ndf_ggx, _lambda_ggx, _masking_smith
  File "I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\ops.py", line 61, in <module>
    torch.utils.cpp_extension.load(name='renderutils_plugin', sources=source_paths, extra_ldflags=ldflags, with_cuda=True, verbose=True)
  File "C:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 1092, in load
    keep_intermediates=keep_intermediates)
  File "C:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 1303, in _jit_compile
    is_standalone=is_standalone)
  File "C:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 1408, in _write_ninja_file_and_build_library
    error_prefix=f"Error building extension '{name}'")
  File "C:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 1682, in _run_ninja_build
    raise RuntimeError(message) from e
RuntimeError: Error building extension 'renderutils_plugin'

Proposition for more comprehensive Readme

First of all, thank you for making this freely available for everyone! This is really a great move!

I wanted to suggest a few things for the readme: 1.) Add "Microsoft Visual C++" to the requirements. It should be clear that you have to tick the C++ development environment while installing VS2019.

2.) Install Git via "conda install git" (This makes it easier to install nvdiffrast and clone the repo)

3.) Change "Install NVDiffRast" to something more descriptive: pip install -e git+https://github.com/NVlabs/nvdiffrast.git@main#egg=nvdiffrast

4.) Add another step to the installation, detailing to move to a desired folder via the Anaconda Prompt and cloning this repo: cd YOUR/PATH/HERE git clone https://github.com/NVlabs/nvdiffmodeling.git and afterwards to move into the folder cd nvdiffmodeling

5.) Installing CUDA via Anaconda only deploys the bare essential files. The CUDA path is not set when doing it this way. So I downloaded the CUDA toolkit from nvidia and installed it afterwards, to fix that.

I am sure that all of these things are obvious for professionals, but it took me a while to figure all of this out, so I would like to save other people's time.

opened by morph3us-net 7
About the raymarch fitting case

Thanks for the marvelous work! I have some questions related to the shadertoy fitting case: It seems figure 22 & figure 23 use some special uv un-wrap methods, can you elaborate the algorithm under the hood?

opened by outlawever 4

RuntimeError: Error building extension 'renderutils_plugin'

opened by TheAlexPG 4

about effect Fig.22 and Fig 23

I read your paper and video in detail，about LEARNING MESH AND MATERIALS FROM IMPLICIT SURFACES,How to improve the effect？ Can you give me some suggestions on setting super parameters? I used my own model as the target, started from the sphere you provided, and carried out experiments. I found that the feet and face were always unsatisfactory? There is a big gap with the rendering of snails and elephants mentioned in your paper. can you give me some adivce.

opened by Lucklycat 2
Texture maps have a lot of noise

Thank you very much for your work！ When I tried the ewer example, I found the following textures： When you zoom in, you will find many red, yellow and green noises This is still the case when I import the output obj into blender, but the contrast image rendered from nvdiffast does not show this situation. What is the reason?

opened by Lucklycat 2
How to improve the optimization effect

Your work is great. Thank you very much for your openness. I tried to use rendering on my own model, but I encountered the following problems and would like to consult: 1.In the process of rendering, I found that the face of the character, especially the eyes, was not poorly optimized, and the color could not be reflected. Are there any parameters that need to be adjusted? 2.During rendering, I encountered that. mtl contains a map_ The texture of d cannot be represented in the renderer. How can I solve this problem?

Thank you for your reply

opened by Lucklycat 2
Lower training resolution => heavier texture pixelation. But why?
Hi, and thanks for great paper and comprehensive code! Could you please help me understand one thing:

I'm running configs/dancer_displacement.json. Despite "texture_res" being 2048, these 2048 x 2048 textures "learn" huge pixels at lower "train_res":

And this isn't just poor interpolation in Blender, but these are actual large solid blocks (of different scale!) in normal_map_opt and kd_map_opt! Here is a part of texture_n.png from the top left experiment, 1:1 scale:

Why do these maps get more pixelated instead of getting more blurry? I've spent couple days searching the code for the source of this behaviour, and I believe there are only three relevant lines (for now let's consider texture only):

rasterization;

interpolating UV from barycentric coordinates;

sampling texture from UV.

Interpolation doesn't use 'nearest' method so it shouldn't be the source of big pixels. And because views are always different, rasterization should be a problem too. But probably I'm just missing some rendering subtleties?

Thanks again!
opened by shrubb 2
msvcrt.lib(loadcfg.obj) : error LNK2001: unresolved external symbol __enclave_config

hi, Thanks for your project. When I try to run sphere to cow demo, I will report this error: msvcrt.lib(loadcfg.obj) : error LNK2001: unresolved external symbol __enclave_config renderutils_plugin.pyd : fatal error LNK1120: Unresolved external command ninja: build stopped: subcommand failed.

Is it because of my Nanjia version? my current version is: nanjia:1.10.2 pytorch:1.8.1 torchvision:0.9.1 cudnn:7.6.5 cuda:10.2

opened by myshiop 2
matrix gradients does not flow back when using renderutils xfm_points

hi, when using renderutils.xfm_points (e.g. in render.py v_pos_clip = ru.xfm_points(mesh.v_pos[None, ...], mtx_in) - and trying to optimize camera pose in mtx_in - gradients does not flow back. Only when using use_python=True it worked.

opened by maorp 1
using camera calibration matrices

Hi, first thanks for sharing this work! I'm trying to solve inverse rendering problem using a photogrammetry dataset which comes with accurate camera calibration mats. i have started from the spot example and i need to convert my intrinsic\extrinsic params to be used in the rastering and cant find an example or a guide to do it. the only hint is that I know it should be converted to openGL and NDC according to nvdiffrast, ill be happy to get an advice, thanks.

opened by maorp 1
some question about the complex material, such as that ray marching snail

Hi, Thank you very much for your research. I'm trying to use your framework to approximate complex materials, but the blender complex material dataset I made with my own is not very good, because I encountered many problems with camera and light settings in my own rendering, Is the ray marching snail used in this paper directly rendered with shadertoy, or is the ray marching process written in Python?

opened by myshiop 1
where can i find the Animation and Skinning demo

hi, i am read the papaer--Appearance-Driven Automatic 3D Model Simplification, think it's great. But in the demo,i can not find the Animation and Skinning demo, where can i find it ? I want to study how to do it. please help ! thanks

opened by Lucklycat 2
CUDA out of memory.

RuntimeError: CUDA out of memory. Tried to allocate 130.00 MiB (GPU 0; 8.00 GiB total capacity; 7.11 GiB already allocated; 0 bytes free; 7.30 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

opened by d-sharafeldeen 2
How to test on custom .obj file?

I am trying to test model simplification on my own .obj. I have created a .mtl file for the .obj but I got an error. How to export .obj file to train the model?

opened by sanbuddhacharyas 0

Unable to execute the program, error during cpp_extension.py running at start

(dmodel) D:\NVIDIATools\nvdiffmodeling-main>python train.py --config configs/spot.json
No CUDA runtime is found, using CUDA_HOME='C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5'
Using C:\Users\redacted\AppData\Local\torch_extensions\torch_extensions\Cache\py36_cpu as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file C:\Users\redacted\AppData\Local\torch_extensions\torch_extensions\Cache\py36_cpu\renderutils_plugin\build.ninja...
Traceback (most recent call last):
  File "train.py", line 20, in <module>
    import src.renderutils as ru
  File "D:\NVIDIATools\nvdiffmodeling-main\src\renderutils\__init__.py", line 9, in <module>
    from .ops import xfm_points, xfm_vectors, image_loss, prepare_shading_normal, lambert, pbr_specular, pbr_bsdf, _fresnel_shlick, _ndf_ggx, _lambda_ggx, _masking_smith
  File "D:\NVIDIATools\nvdiffmodeling-main\src\renderutils\ops.py", line 61, in <module>
    torch.utils.cpp_extension.load(name='renderutils_plugin', sources=source_paths, extra_ldflags=ldflags, with_cuda=True, verbose=True)
  File "C:\Users\redacted\anaconda3\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 1136, in load
    keep_intermediates=keep_intermediates)
  File "C:\Users\redacted\anaconda3\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 1347, in _jit_compile
    is_standalone=is_standalone)
  File "C:\Users\redacted\anaconda3\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 1445, in _write_ninja_file_and_build_library
    is_standalone=is_standalone)
  File "C:\Users\redacted\anaconda3\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 1834, in _write_ninja_file_to_build_library
    cuda_flags = common_cflags + COMMON_NVCC_FLAGS + _get_cuda_arch_flags()
  File "C:\Users\redacted\anaconda3\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 1606, in _get_cuda_arch_flags
    arch_list[-1] += '+PTX'
IndexError: list index out of range

(dmodel) D:\NVIDIATools\nvdiffmodeling-main>

After installing the prerequisites, only difference to the instruction is CUDA 11.5 (and i changed the version to the pip install line to that same version) Getting the above when attempting to run the example spot cow training.

Something i'm doing wrong, incompatibility with versions or something else entirely?

opened by RiversJohn 3

Differentiable rasterization applied to 3D model simplification tasks

Related tags

Overview

nvdiffmodeling

License

Installation

One time setup (Windows)

Every new command prompt

Examples

Server usage (through Docker)

Comments

Releases(v1.0)

v1.0(Apr 13, 2021)

Owner

NVIDIA Research Projects

[CVPR 21] Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.

Code and models used in "MUSS Multilingual Unsupervised Sentence Simplification by Mining Paraphrases".

Differentiable Neural Computers, Sparse Access Memory and Sparse Differentiable Neural Computers, for Pytorch

Ejemplo Algoritmo Viterbi - Example of a Viterbi algorithm applied to a hidden Markov model on DNA sequence

DiffQ performs differentiable quantization using pseudo quantization noise. It can automatically tune the number of bits used per weight or group of weights, in order to achieve a given trade-off between model size and accuracy.

Pytorch Implementation of Google's Parallel Tacotron 2: A Non-Autoregressive Neural TTS Model with Differentiable Duration Modeling

Bachelor's Thesis in Computer Science: Privacy-Preserving Federated Learning Applied to Decentralized Data

Framework that uses artificial intelligence applied to mathematical models to make predictions

nnDetection is a self-configuring framework for 3D (volumetric) medical object detection which can be applied to new data sets without manual intervention. It includes guides for 12 data sets that were used to develop and evaluate the performance of the proposed method.

We present a framework for training multi-modal deep learning models on unlabelled video data by forcing the network to learn invariances to transformations applied to both the audio and video streams.

Tensorforce: a TensorFlow library for applied reinforcement learning

Neural networks applied in recognizing guitar chords using python, AutoML.NET with C# and .NET Core

Object tracking and object detection is applied to track golf puts in real time and display stats/games.

Deep Learning applied to Integral data analysis

This is the code for our KILT leaderboard submission to the T-REx and zsRE tasks. It includes code for training a DPR model then continuing training with RAG.

gym-anm is a framework for designing reinforcement learning (RL) environments that model Active Network Management (ANM) tasks in electricity distribution networks.

A lightweight Python-based 3D network multi-agent simulator. Uses a cell-based congestion model. Calculates risk, loudness and battery capacities of the agents. Suitable for 3D network optimization tasks.

In this project we investigate the performance of the SetCon model on realistic video footage. Therefore, we implemented the model in PyTorch and tested the model on two example videos.

Step by Step on how to create an vision recognition model using LOBE.ai, export the model and run the model in an Azure Function