Hello Yan @traveller59 , I am trying to deploy models with spconv layers (tensorRT) and found I may need to dig into pytorch source codes. I am wondering if there is any easier way for deploying models with spconv such that I could directly utilize c++ for fast inference?

Thanks in advance.

Thanks for your excellent work!

I am trying to install Spconv 2.0 with the latest pytorch 1.10.0 and Cuda 11.4 or Cuda 11.5. However, I meet different errors when I try to install Spconv with pip install spconv-cu114 or building from source.

If I use pip install spconv-cu114, when I try to run import spconv.pytorch as spconv, I met the following error message:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/du014/anaconda3/envs/pointcloud-mk2/lib/python3.8/site-packages/spconv/pytorch/__init__.py", line 8, in <module>
    from spconv.pytorch.conv import (SparseConv2d, SparseConv3d, SparseConvTranspose2d,
  File "/home/du014/anaconda3/envs/pointcloud-mk2/lib/python3.8/site-packages/spconv/pytorch/conv.py", line 29, in <module>
    from spconv.pytorch.core import IndiceData, SparseConvTensor
  File "/home/du014/anaconda3/envs/pointcloud-mk2/lib/python3.8/site-packages/spconv/pytorch/core.py", line 17, in <module>
    from .core_fx import *
  File "/home/du014/anaconda3/envs/pointcloud-mk2/lib/python3.8/site-packages/spconv/pytorch/core_fx.py", line 20, in <module>
    from torch.fx.symbolic_trace import ProxyableClassMeta
ModuleNotFoundError: No module named 'torch.fx.symbolic_trace'

But when I turn to building spconv from source, I met another import error as following:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/du014/anaconda3/envs/pointcloud-mk2/lib/python3.8/site-packages/spconv/pytorch/__init__.py", line 7, in <module>
    from spconv.pytorch import ops
  File "/home/du014/anaconda3/envs/pointcloud-mk2/lib/python3.8/site-packages/spconv/pytorch/ops.py", line 22, in <module>
    from spconv.algo import AlgoHint, ConvAlgo
  File "/home/du014/anaconda3/envs/pointcloud-mk2/lib/python3.8/site-packages/spconv/algo.py", line 18, in <module>
    from spconv.core_cc.cumm.gemm.main import GemmAlgoDesp, GemmMainUnitTest, GemmParams
ModuleNotFoundError: No module named 'spconv.core_cc'

BTW, I have already install cum-cu114.

Could you help me with this import bug?

Thanks for your work, now I counter the problem below when training during the backward

Traceback (most recent call last): File "./src/main.py", line 105, in main(opt) File "./src/main.py", line 73, in main log_dict_train, _ = trainer.train(epoch, train_loader) File "/dockerdata/hongliding/codes/pole-3d/src/lib/trains/base_trainer.py", line 136, in train return self.run_epoch('train', epoch, data_loader) File "/dockerdata/hongliding/codes/pole-3d/src/lib/trains/base_trainer.py", line 85, in run_epoch loss.backward() File "/dockerdata/hongliding/env/sp2py3.6/lib/python3.6/site-packages/torch/tensor.py", line 185, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph) File "/dockerdata/hongliding/env/sp2py3.6/lib/python3.6/site-packages/torch/autograd/init.py", line 127, in backward allow_unreachable=True) # allow_unreachable flag RuntimeError: /io/build/temp.linux-x86_64-3.6/spconv/build/src/cumm/conv/main/ConvMainUnitTest/ConvMainUnitTest_implicit_gemm2.cu 3187 cuda execution failed with error 700 an illegal memory access was encountered ???

my configuration is: cuda 10.2, spconv 2.1.12, Gpu v100, pytorch 1.6.0, python 3.6.8

Make sure you are running the following commands on a host with a Pascal GPU. (Newer architectures probably also works.)

conda create --name spconv-1.0 python=3.6 pytorch=1.1 cudatoolkit=9.2 cudatoolkit-dev=9.2 \
cmake --channel pytorch --channel=conda-forge
conda activate spconv-1.0
conda install cudnn
conda install boost
git clone https://github.com/traveller59/spconv spconv_8da6f96 --recursive
cd spconv_8da6f96
git checkout 8da6f967fb9a054d8870c3515b1b44eca2103634

If necessary: Download and extract 0001-Allow-to-specifiy-CUDA_ROOT-directory-and-pick-corre.patch.zip and patch spconv via: git am <PATH_TO_EXTRACTED_FILE>/0001-Allow-to-specifiy-CUDA_ROOT-directory-and-pick-corre.patch

(or fix issue #78 by yourself)

CUDA_ROOT=<PATH_TO_YOUR_CONDA_INSTALLATION>/conda_envs/spconv-1.0 python setup.py bdist_wheel
cd dist/
pip install *

Test spconv via: python -c 'import spconv' (should just return and not raise any errors)

Hello! Thanks for open-sourcing this amazing repository. I got several fundamental questions,

1. Under my understanding, spconv is actually another implementation of SparseConvNet, so basically there is not much difference between these two implementations right?
2. The SparseConv3d is a regular convolution definition, but for SubMConv3d, the convolution output will be counted only when the kernel center covers an input site. So I don't understand how the param stride affects the output in SubMConv3d, or based on my understanding, there is actually no stride in SubMConv3d?
3. Could you kindly offer a code example for something like UNet(encoder-decoder)? I found that after I stack SparseConv3d and SparseConvTranspose3d the resolution of the point cloud just can't recover.

Hoping to get your answers!

1. voxelization generation
2. max/avg pool
3. normal sparse conv unrolling
4. gather/scatter time benchmark, fused gemm by hand(Tiled Matrix Multiplication by CUDA) or fused gemm by CUTLASS or any other opt by CUTLASS

1 is the main bottleneck currently. 1 and 2 could be optimized together. 3 is the bottleneck in conv, aka, stride=2 normal sparse conv. 4 is not bottleneck in spconv. Further optimization need benchmark further.

According to Cannikin Law, we should do 1,2 first, which is easy to implement by Atomic operation.

My CUDA-9.0 is installed at /scratch_net/hox/mhahner/apps/cuda-9.0 because I have no write permissions at /usr/local

Where is the CUDA path assumed? How can I fix this?

The script already found my CUDA installation path correctly, see -- Found CUDA: /scratch_net/hox/mhahner/apps/cuda-9.0 (found version "9.0") but still raises the error: make[2]: *** No rule to make target '/usr/local/cuda/lib64/libnvToolsExt.so', needed by '../lib.linux-x86_64-3.7/spconv/libspconv.so'. Stop.

Here are the full details:

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 9.9 (stretch)
Release:	9.9
Codename:	stretch

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Sep__1_21:08:03_CDT_2017
Cuda compilation tools, release 9.0, V9.0.176

$ cat $CUDA_ROOT/include/cudnn.h | grep CUDNN_MAJOR -A 2
#define CUDNN_MAJOR 7
#define CUDNN_MINOR 1
#define CUDNN_PATCHLEVEL 3

$ cmake --version
cmake version 3.15.2

$ cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module  390.116  Sun Jan 27 07:21:36 PST 2019
GCC version:  gcc version 6.3.0 20170516 (Debian 6.3.0-18+deb9u1)

$ python
Python 3.7.4 (default, Aug 13 2019, 20:35:49) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> print(torch.__version__)
1.2.0

$ echo $CUDA_ROOT
/scratch_net/hox/mhahner/apps/cuda-9.0

$ echo $CUDA_TOOLKIT_ROOT
/scratch_net/hox/mhahner/apps/cuda-9.0/lib64

$ echo $CMAKE_CUDA_COMPILER
/scratch_net/hox/mhahner/apps/cuda-9.0/bin/nvcc

$ python setup.py bdist_wheel
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.7
creating build/lib.linux-x86_64-3.7/spconv
copying spconv/functional.py -> build/lib.linux-x86_64-3.7/spconv
copying spconv/ops.py -> build/lib.linux-x86_64-3.7/spconv
copying spconv/pool.py -> build/lib.linux-x86_64-3.7/spconv
copying spconv/modules.py -> build/lib.linux-x86_64-3.7/spconv
copying spconv/conv.py -> build/lib.linux-x86_64-3.7/spconv
copying spconv/test_utils.py -> build/lib.linux-x86_64-3.7/spconv
copying spconv/__init__.py -> build/lib.linux-x86_64-3.7/spconv
creating build/lib.linux-x86_64-3.7/spconv/utils
copying spconv/utils/__init__.py -> build/lib.linux-x86_64-3.7/spconv/utils
running build_ext
Release
|||||CMAKE ARGS||||| ['-DCMAKE_PREFIX_PATH=/home/mhahner/scratch/apps/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch', '-DPYBIND11_PYTHON_VERSION=3.7', '-DSPCONV_BuildTests=OFF', '-DSPCONV_BuildCUDA=OFF', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/scratch_net/hox/mhahner/polybox/PhD/repositories/spconv/build/lib.linux-x86_64-3.7/spconv', '-DCMAKE_BUILD_TYPE=Release']
-- The CXX compiler identification is GNU 6.3.0
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for C++ include pthread.h
-- Looking for C++ include pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Found CUDA: /scratch_net/hox/mhahner/apps/cuda-9.0 (found version "9.0") 
-- Caffe2: CUDA detected: 9.0
-- Caffe2: CUDA nvcc is: /scratch_net/hox/mhahner/apps/cuda-9.0/bin/nvcc
-- Caffe2: CUDA toolkit directory: /scratch_net/hox/mhahner/apps/cuda-9.0
-- Caffe2: Header version is: 9.0
-- Found CUDNN: /scratch_net/hox/mhahner/apps/cuda-9.0/include  
-- Found cuDNN: v7.1.3  (include: /scratch_net/hox/mhahner/apps/cuda-9.0/include, library: /scratch_net/hox/mhahner/apps/cuda-9.0/lib64/libcudnn.so)
-- Autodetected CUDA architecture(s):  3.5
-- Added CUDA NVCC flags for: -gencode;arch=compute_35,code=sm_35
-- Found torch: /home/mhahner/scratch/apps/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/lib/libtorch.so  
-- Found PythonInterp: /home/mhahner/scratch/apps/anaconda3/envs/pytorch/bin/python3.7 (found suitable version "3.7.4", minimum required is "3.7") 
-- Found PythonLibs: /home/mhahner/scratch/apps/anaconda3/envs/pytorch/lib/libpython3.7m.so
-- Performing Test HAS_CPP14_FLAG
-- Performing Test HAS_CPP14_FLAG - Success
-- pybind11 v2.3.dev0
-- Found OpenMP_CXX: -fopenmp (found version "4.5") 
-- Found OpenMP: TRUE (found version "4.5")  
-- Configuring done
-- Generating done
-- Build files have been written to: /scratch_net/hox/mhahner/polybox/PhD/repositories/spconv/build/temp.linux-x86_64-3.7
Scanning dependencies of target spconv_utils
Scanning dependencies of target spconv
[ 12%] Building CXX object src/utils/CMakeFiles/spconv_utils.dir/all.cc.o
[ 25%] Building CXX object src/spconv/CMakeFiles/spconv.dir/all.cc.o
[ 37%] Building CXX object src/spconv/CMakeFiles/spconv.dir/indice.cc.o
[ 50%] Building CXX object src/spconv/CMakeFiles/spconv.dir/reordering.cc.o
[ 62%] Building CXX object src/spconv/CMakeFiles/spconv.dir/maxpool.cc.o
[ 75%] Building CXX object src/spconv/CMakeFiles/spconv.dir/nms.cc.o
make[2]: *** No rule to make target '/usr/local/cuda/lib64/libnvToolsExt.so', needed by '../lib.linux-x86_64-3.7/spconv/libspconv.so'.  Stop.
make[2]: *** Waiting for unfinished jobs....
[ 87%] Linking CXX shared library ../../../lib.linux-x86_64-3.7/spconv/spconv_utils.cpython-37m-x86_64-linux-gnu.so
[ 87%] Built target spconv_utils
CMakeFiles/Makefile2:117: recipe for target 'src/spconv/CMakeFiles/spconv.dir/all' failed
make[1]: *** [src/spconv/CMakeFiles/spconv.dir/all] Error 2
Makefile:129: recipe for target 'all' failed
make: *** [all] Error 2
Traceback (most recent call last):
  File "setup.py", line 99, in <module>
    zip_safe=False,
  File "/home/mhahner/scratch/apps/anaconda3/envs/pytorch/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup
    return distutils.core.setup(**attrs)
  File "/home/mhahner/scratch/apps/anaconda3/envs/pytorch/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/home/mhahner/scratch/apps/anaconda3/envs/pytorch/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/home/mhahner/scratch/apps/anaconda3/envs/pytorch/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/mhahner/scratch/apps/anaconda3/envs/pytorch/lib/python3.7/site-packages/wheel/bdist_wheel.py", line 192, in run
    self.run_command('build')
  File "/home/mhahner/scratch/apps/anaconda3/envs/pytorch/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/home/mhahner/scratch/apps/anaconda3/envs/pytorch/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/mhahner/scratch/apps/anaconda3/envs/pytorch/lib/python3.7/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/home/mhahner/scratch/apps/anaconda3/envs/pytorch/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/home/mhahner/scratch/apps/anaconda3/envs/pytorch/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "setup.py", line 40, in run
    self.build_extension(ext)
  File "setup.py", line 83, in build_extension
    subprocess.check_call(['cmake', '--build', '.'] + build_args, cwd=self.build_temp)
  File "/home/mhahner/scratch/apps/anaconda3/envs/pytorch/lib/python3.7/subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--config', 'Release', '--', '-j4']' returned non-zero exit status 2.

First of all, thank you very much for your code.I have a problem when running the command python setup.py bdist_wheel .Although I followed the instructions in #78 and #70 by @MartinHahner, #21, #167 , it did not work.Then I rummaged through almost all the problems, but I didn’t solve it.I would appreciate it if you can help me.My environment is as follows: OS version：Ubuntu 18.04 GPU name：TITAN V CUDA version：9.2 Pytorch：1.1 Python：3.6 gcc version：7.5.0

running bdist_wheel running build running build_py running build_ext /home/gye/PointCloud/objectDetection/spconv_8da6f96/build/lib.linux-x86_64-3.6 Release -- The CUDA compiler identification is unknown -- Check for working CUDA compiler: /home/gye/anaconda3/envs/cuda92/bin/nvcc -- Check for working CUDA compiler: /home/gye/anaconda3/envs/cuda92/bin/nvcc -- broken CMake Error at /home/gye/.local/lib/python3.6/site-packages/cmake/data/share/cmake-3.13/Modules/CMakeTestCUDACompiler.cmake:46 (message): The CUDA compiler

"/home/gye/anaconda3/envs/cuda92/bin/nvcc"

is not able to compile a simple test program.

It fails with the following output:

Change Dir: /home/gye/PointCloud/objectDetection/spconv_8da6f96/build/temp.linux-x86_64-3.6/CMakeFiles/CMakeTmp

Run Build Command:"/usr/bin/make" "cmTC_cd192/fast"
/usr/bin/make -f CMakeFiles/cmTC_cd192.dir/build.make CMakeFiles/cmTC_cd192.dir/build
make[1]: 进入目录“/home/gye/PointCloud/objectDetection/spconv_8da6f96/build/temp.linux-x86_64-3.6/CMakeFiles/CMakeTmp”
Building CUDA object CMakeFiles/cmTC_cd192.dir/main.cu.o
/home/gye/anaconda3/envs/cuda92/bin/nvcc    "--expt-relaxed-constexpr"    -x cu -c /home/gye/PointCloud/objectDetection/spconv_8da6f96/build/temp.linux-x86_64-3.6/CMakeFiles/CMakeTmp/main.cu -o CMakeFiles/cmTC_cd192.dir/main.cu.o
Linking CUDA device code CMakeFiles/cmTC_cd192.dir/cmake_device_link.o
/home/gye/.local/lib/python3.6/site-packages/cmake/data/bin/cmake -E cmake_link_script CMakeFiles/cmTC_cd192.dir/dlink.txt --verbose=1
/home/gye/anaconda3/envs/cuda92/bin/nvcc  "--expt-relaxed-constexpr"    -shared -dlink CMakeFiles/cmTC_cd192.dir/main.cu.o -o CMakeFiles/cmTC_cd192.dir/cmake_device_link.o 
Linking CUDA executable cmTC_cd192
/home/gye/.local/lib/python3.6/site-packages/cmake/data/bin/cmake -E cmake_link_script CMakeFiles/cmTC_cd192.dir/link.txt --verbose=1
""   CMakeFiles/cmTC_cd192.dir/main.cu.o CMakeFiles/cmTC_cd192.dir/cmake_device_link.o -o cmTC_cd192 
Error running link command: No such file or directory
CMakeFiles/cmTC_cd192.dir/build.make:105: recipe for target 'cmTC_cd192' failed
make[1]: *** [cmTC_cd192] Error 2
make[1]: 离开目录“/home/gye/PointCloud/objectDetection/spconv_8da6f96/build/temp.linux-x86_64-3.6/CMakeFiles/CMakeTmp”
Makefile:121: recipe for target 'cmTC_cd192/fast' failed
make: *** [cmTC_cd192/fast] Error 2

CMake will not be able to correctly generate this project. Call Stack (most recent call first): CMakeLists.txt:2 (project)

-- Configuring incomplete, errors occurred! See also "/home/gye/PointCloud/objectDetection/spconv_8da6f96/build/temp.linux-x86_64-3.6/CMakeFiles/CMakeOutput.log". See also "/home/gye/PointCloud/objectDetection/spconv_8da6f96/build/temp.linux-x86_64-3.6/CMakeFiles/CMakeError.log". Traceback (most recent call last): File "setup.py", line 103, in zip_safe=False, File "/home/gye/anaconda3/envs/spconv/lib/python3.6/site-packages/setuptools/init.py", line 163, in setup return distutils.core.setup(**attrs) File "/home/gye/anaconda3/envs/spconv/lib/python3.6/distutils/core.py", line 148, in setup dist.run_commands() File "/home/gye/anaconda3/envs/spconv/lib/python3.6/distutils/dist.py", line 955, in run_commands self.run_command(cmd) File "/home/gye/anaconda3/envs/spconv/lib/python3.6/distutils/dist.py", line 974, in run_command cmd_obj.run() File "/home/gye/anaconda3/envs/spconv/lib/python3.6/site-packages/wheel/bdist_wheel.py", line 290, in run self.run_command('build') File "/home/gye/anaconda3/envs/spconv/lib/python3.6/distutils/cmd.py", line 313, in run_command self.distribution.run_command(command) File "/home/gye/anaconda3/envs/spconv/lib/python3.6/distutils/dist.py", line 974, in run_command cmd_obj.run() File "/home/gye/anaconda3/envs/spconv/lib/python3.6/distutils/command/build.py", line 135, in run self.run_command(cmd_name) File "/home/gye/anaconda3/envs/spconv/lib/python3.6/distutils/cmd.py", line 313, in run_command self.distribution.run_command(command) File "/home/gye/anaconda3/envs/spconv/lib/python3.6/distutils/dist.py", line 974, in run_command cmd_obj.run() File "setup.py", line 39, in run self.build_extension(ext) File "setup.py", line 71, in build_extension subprocess.check_call(['cmake', ext.sourcedir] + cmake_args, cwd=self.build_temp, env=env) File "/home/gye/anaconda3/envs/spconv/lib/python3.6/subprocess.py", line 311, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['cmake', '/home/gye/PointCloud/objectDetection/spconv_8da6f96', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/home/gye/PointCloud/objectDetection/spconv_8da6f96/build/lib.linux-x86_64-3.6/spconv', '-DCMAKE_PREFIX_PATH=/home/gye/anaconda3/envs/spconv/lib/python3.6/site-packages/torch', '-DPYBIND11_PYTHON_VERSION=3.6', '-DSPCONV_BuildTests=OFF', '-DCMAKE_CUDA_FLAGS="--expt-relaxed-constexpr"', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_CUDA_COMPILER:FILEPATH=/home/gye/anaconda3/envs/cuda92/bin/nvcc']' returned non-zero exit status 1.

Hello,

I occasionally got the following error when using spconv. It almost works well when I disconnect my internet connection.

[33750/39521]
cudahash: Completely failed to build
Cuda error in file '/xxxxxx/spconv/src/cuhash/hash_table.cpp' in line 194 : an illegal memory access was encountered.

Any advice for this error?

By the way, I'm planning to wrap this repo into a TensorFlow version. Hope for further help here.

Environment: GPU RTX2080TI Cuda driver: 465.31 spconv-cu111: 2.1.16 torch: 1.9.1

When using spconv1 everything works fine (also when using ConvAlgo.Native), but when using ConvAlgo.MaskImplicitGemm (default algo) I get a runtime error thrown from SpconvOps.sort_1d_by_key_allocator

Am getting the error when doing forward of SubMConv3d, managed to narrow it down to the following function (with args to reproduce):

get_indice_pairs_implicit_gemm(x.indices[:2], batch_size=x.batch_size, spatial_shape=x.spatial_shape, algo=ConvAlgo.MaskSplitImplicitGemm, ksize=(1, 1, 1), stride=(1,1,1), padding=(0,1,1), dilation=(0,0,0), out_padding=(0,0,0), subm=True, transpose=False, is_train=False, alloc=x.thrust_allocator, timer=x._timer)

x.indices[:2]=tensor([[ 0, 9, 54, 19], [ 0, 9, 55, 19]], device='cuda:0', dtype=torch.int32) x.batch_size=1, x.spatial_shape=[480 360 32], x.thrust_allocator=None, x._timer=<spconv.tools.CUDAKernelTimer object at 0x7f6de080c220>)

  Index numValidPoints = 0;
  Index validPoints[KernelMaxVolume * (NDim + 1)];
  Index *pointPtr = nullptr;
  Index index = 0;
  for (int ix : tv::KernelLoopX<int>(numActIn)) {
    numValidPoints = getValidOutPos<Index, NDim>(
        indicesIn.data() + ix * (NDim + 1) + 1, kernelSize.data(),
        stride.data(), padding.data(), dilation.data(), outSpatialShape.data(),
        validPoints);
    for (int i = 0; i < numValidPoints; ++i) {
      pointPtr = validPoints + i * (NDim + 1);
      auto offset = pointPtr[NDim];
      index = tv::ArrayIndexRowMajor<NDim, NDim>::runPtrs(
                  pointPtr, outSpatialShape.data(), 0) +
              spatialVolume * indicesIn(ix, 0);
      if (gridsOut[index] > -1) {
        Index oldNum = atomicAdd(indiceNum.data() + offset, Index(1));
        indicePairs(1, offset, oldNum) = gridsOut[index];
        indicePairs(0, offset, oldNum) = ix;
      }
    }
  }

This is the code snip of indice pair generation, my question is about the variable validPoints. CUDA kernel reuse the same validPoints.

Isn't there have data access conflicting?

By the way, what about change KernelMaxVolume to KernelVolume? What will happen after this change?

v1.2.1 (spconv) azuryl@delight-gpu:~/project/spconv$ python setup.py bdist_wheel running bdist_wheel running build running build_ext Release |||||CMAKE ARGS||||| ['-DCMAKE_PREFIX_PATH=/home/azuryl/anaconda3/envs/spconv/lib/python3.8/site-packages/torch', '-DPYBIND11_PYTHON_VERSION=3.8', '-DSPCONV_BuildTests=OFF', '-DPYTORCH_VERSION=11200', '-DCMAKE_CUDA_FLAGS="--expt-relaxed-constexpr" -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/home/azuryl/project/spconv/build/lib.linux-x86_64-cpython-38/spconv', '-DCMAKE_BUILD_TYPE=Release'] -- Caffe2: CUDA detected: 10.2 -- Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc -- Caffe2: CUDA toolkit directory: /usr/local/cuda -- Caffe2: Header version is: 10.2 -- Found cuDNN: v8.7.0 (include: /usr/local/cuda/include, library: /usr/local/cuda/lib64/libcudnn.so) -- /usr/local/cuda/lib64/libnvrtc.so shorthash is 08c4863f -- Autodetected CUDA architecture(s): 7.5 7.5 7.5 7.5 7.5 7.5 7.5 7.5 -- Added CUDA NVCC flags for: -gencode;arch=compute_75,code=sm_75 CMake Warning at /home/azuryl/anaconda3/envs/spconv/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:22 (message): static library kineto_LIBRARY-NOTFOUND not found. Call Stack (most recent call first): /home/azuryl/anaconda3/envs/spconv/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:127 (append_torchlib_if_found) CMakeLists.txt:22 (find_package)

-- Autodetected CUDA architecture(s): 7.5 7.5 7.5 7.5 7.5 7.5 7.5 7.5 CMake Error at CMakeLists.txt:41 (add_subdirectory): The source directory

/home/azuryl/project/spconv/third_party/pybind11

does not contain a CMakeLists.txt file.

-- Configuring incomplete, errors occurred! See also "/home/azuryl/project/spconv/build/temp.linux-x86_64-cpython-38/CMakeFiles/CMakeOutput.log". See also "/home/azuryl/project/spconv/build/temp.linux-x86_64-cpython-38/CMakeFiles/CMakeError.log". Traceback (most recent call last): File "setup.py", line 96, in setup( File "/home/azuryl/anaconda3/envs/spconv/lib/python3.8/site-packages/setuptools/init.py", line 87, in setup return distutils.core.setup(**attrs) File "/home/azuryl/anaconda3/envs/spconv/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 185, in setup return run_commands(dist) File "/home/azuryl/anaconda3/envs/spconv/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 201, in run_commands dist.run_commands() File "/home/azuryl/anaconda3/envs/spconv/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 968, in run_commands self.run_command(cmd) File "/home/azuryl/anaconda3/envs/spconv/lib/python3.8/site-packages/setuptools/dist.py", line 1217, in run_command super().run_command(command) File "/home/azuryl/anaconda3/envs/spconv/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 987, in run_command cmd_obj.run() File "/home/azuryl/anaconda3/envs/spconv/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 299, in run self.run_command('build') File "/home/azuryl/anaconda3/envs/spconv/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command self.distribution.run_command(command) File "/home/azuryl/anaconda3/envs/spconv/lib/python3.8/site-packages/setuptools/dist.py", line 1217, in run_command super().run_command(command) File "/home/azuryl/anaconda3/envs/spconv/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 987, in run_command cmd_obj.run() File "/home/azuryl/anaconda3/envs/spconv/lib/python3.8/site-packages/setuptools/_distutils/command/build.py", line 132, in run self.run_command(cmd_name) File "/home/azuryl/anaconda3/envs/spconv/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command self.distribution.run_command(command) File "/home/azuryl/anaconda3/envs/spconv/lib/python3.8/site-packages/setuptools/dist.py", line 1217, in run_command super().run_command(command) File "/home/azuryl/anaconda3/envs/spconv/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 987, in run_command cmd_obj.run() File "setup.py", line 48, in run self.build_extension(ext) File "setup.py", line 91, in build_extension subprocess.check_call(['cmake', ext.sourcedir] + cmake_args, cwd=self.build_temp, env=env) File "/home/azuryl/anaconda3/envs/spconv/lib/python3.8/subprocess.py", line 364, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['cmake', '/home/azuryl/project/spconv', '-DCMAKE_PREFIX_PATH=/home/azuryl/anaconda3/envs/spconv/lib/python3.8/site-packages/torch', '-DPYBIND11_PYTHON_VERSION=3.8', '-DSPCONV_BuildTests=OFF', '-DPYTORCH_VERSION=11200', '-DCMAKE_CUDA_FLAGS="--expt-relaxed-constexpr" -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/home/azuryl/project/spconv/build/lib.linux-x86_64-cpython-38/spconv', '-DCMAKE_BUILD_TYPE=Release']' returned non-zero exit status 1. (spconv) azuryl@delight-gpu:~/project/spconv$

after running function spconv/pytorch/ops.py:SpconvOps.generate_subm_conv_inds, I want to print data of each *_tv. I have opened global variable DEBUG. But the print is always initialized value. How can i print real value of *_tv?

1. In demo SECOND the point_cloud_range is [0, -40, -3, 70.4, 40, 1], voxel_size is [0.05, 0.05, 0.1]. the sparse_shape_z is 41 not 40((1-3)/0.1). Do you know the reason?

2.In function spconv/pytorch/ops.py: get_indice_pairs_implicit_gemm. I see the realization of generate_subm_conv_inds_cpu uses std::unordered_map as hash_table to search the same index of input coor and output coor. But in GPU, I can't understand how to realize the hash search of same index. Have you compare all output coor indices and input coor indices to find the pair with GPU parallel equal function, or other methods?

3. In get_indice_pairs_implicit_gemm, Is the parameter "indices_pair_mask(pair_mask_tv)" and function "sort_1d_by_key_allocator" used for mask_split_implicit_gemm? The SECOND uses only mask_implicit_gemm, in this context, Does the above parameter and function work?

NVIDIA-SMI 470.82.01 Driver Version: 470.82.01 CUDA Version: 11.4

python3 --version Python 3.9.5

import spconv spconv.version '2.2.6'

To reproduce

import spconv.pytorch as spconv
import torch

num_channels = 32
N = 4

features = torch.rand([N, num_channels])
# Coordinates with shape [N, ndim + 1], batch index must be put in indices[:, 0]
indices_np = np.array([[0, 1, 1, 2], [0, 2, 1, 1], [1, 7, 8, 2], [1, 7, 9, 2]]).astype(np.int32)
indices = torch.from_numpy(indices_np)
spatial_shape = [513, 517, 29]
batch_size = 2
device_idx = 0
x = spconv.SparseConvTensor(features.cuda(device_idx), indices.cuda(device_idx), spatial_shape, batch_size)
conv = spconv.SubMConv3d(num_channels, 16, 3, padding=1, bias=False, indice_key="subm1").cuda(device_idx)
print(conv(x))

command CUDA_LAUNCH_BLOCKING=1 python3 /repro.py

If I run with device_idx=0 output is as expected

<spconv.pytorch.core.SparseConvTensor object at 0x7f9f76e15ca0>

If I run with device_idx = 1 output is:

[Exception|implicit_gemm_pair]indices=torch.Size([4, 4]),bs=2,ss=[513, 517, 29],algo=ConvAlgo.MaskImplicitGemm,ksize=[3, 3, 3],stride=[1, 1, 1],padding=[1, 1, 1],dilation=[1, 1, 1],subm=True,transpose=False spconv try to save debug data to /debug.txt, but failed with exception CUDA error: an illegal memory access was encountered. please check your SPCONV_DEBUG_SAVE_PATH Traceback (most recent call last): File "/repro.py", line 17, in <module> print(conv(x)) File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "/usr/local/lib/python3.9/dist-packages/spconv/pytorch/conv.py", line 515, in forward raise e File "/usr/local/lib/python3.9/dist-packages/spconv/pytorch/conv.py", line 492, in forward res = ops.get_indice_pairs_implicit_gemm( File "/usr/local/lib/python3.9/dist-packages/spconv/pytorch/ops.py", line 546, in get_indice_pairs_implicit_gemm SpconvOps.sort_1d_by_key_allocator(pair_mask_tv[j], RuntimeError: radix_sort: failed on 1st step: cudaErrorIllegalAddress: an illegal memory access was encountered

No debug data was saved to the file SPCONV_DEBUG_SAVE_PATH

To me it looks like it is trying to do the rules on GPU0 even though all the tensors on are GPU1.

Note it works fine if we run with with torch.cuda.device(1):

when using spconv.ops.get_indice_pairs_implicit_gemm, the output tuple has pair bwd. It seems the pair that matches between input indices and corresponding indices of output. Am i right??? If it's right, then what's the meaning of -1 of the values of pair_bwd, and how is the order of the indices defined??