Kernel Point Convolutions

Hi @HuguesTHOMAS Sorry for the bothering, have you ever met such kind of problem during training ?

My Configuration is:

CUDA Version 9.0.176, TF1.12,  GTX 1080

Like Issue15, this error also occurs randomly during training.

Hi, Thanks for your sharing. I have tried your code on my own dataset but the I found that initially everything goes well but after several epochs the training suddenly broke up ( accuracy becomes 1 and the loss becomes 0 ) I use tf 1.12.0 and the cuda version is 9.0, cudnn version is 7.1.4

# conda list | grep tensorflow
tensorflow-estimator      1.13.0                     py_0    anaconda
tensorflow-gpu            1.12.0                   pypi_0    pypi
tensorflow-tensorboard    0.4.0                    pypi_0    pypi

Have you met this kind of problem? Another potential problem is that sometimes the training takes 4400 MB GPU memory (see from nvidia-smi), but sometimes it takes more than 7000 MB ( and I do not change the batch size and network architecture) I am pretty confused about these problems. Could you give me some advice?

I am wondering, since the data preparation happens in batch before being sent to the network, whether it would be reasonable to adapt it to run on-demand as a new cloud arrives, or whether the processing of each new input would make it prohibitively costly. Thanks

Hi, @HuguesTHOMAS , Firstly, thanks for your great work on KPConv. Here I have met some problems when I run 'training_S3DIS.py'. The error information is below:

Traceback (most recent call last): File "/home/hwk/anaconda3/envs/py3.5/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1278, in _do_call return fn(*args) File "/home/hwk/anaconda3/envs/py3.5/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1263, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/home/hwk/anaconda3/envs/py3.5/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1350, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence [[Node: IteratorGetNext = IteratorGetNextoutput_shapes=[[?,3], [?,3], [?,3], [?,3], [?,3], ..., [?], [?,3], [?,3,3], [?], [?]], output_types=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_INT32, DT_FLOAT, DT_FLOAT, DT_INT32, DT_INT32], _device="/job:localhost/replica:0/task:0/device:CPU:0"]] [[Node: optimizer/gradients/KernelPointNetwork/layer_0/resnetb_1/conv2/concat_1_grad/GatherV2_2/axis/_222 = _HostSendT=DT_INT32, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1469_...rV2_2/axis", _device="/job:localhost/replica:0/task:0/device:GPU:0"]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/hwk/KPConv/utils/trainer.py", line 261, in train _, L_out, L_reg, L_p, probs, labels, acc = self.sess.run(ops, {model.dropout_prob: 0.5}) File "/home/hwk/anaconda3/envs/py3.5/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 877, in run run_metadata_ptr) File "/home/hwk/anaconda3/envs/py3.5/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1100, in _run feed_dict_tensor, options, run_metadata) File "/home/hwk/anaconda3/envs/py3.5/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1272, in _do_run run_metadata) File "/home/hwk/anaconda3/envs/py3.5/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1291, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence [[Node: IteratorGetNext = IteratorGetNextoutput_shapes=[[?,3], [?,3], [?,3], [?,3], [?,3], ..., [?], [?,3], [?,3,3], [?], [?]], output_types=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_INT32, DT_FLOAT, DT_FLOAT, DT_INT32, DT_INT32], _device="/job:localhost/replica:0/task:0/device:CPU:0"]] [[Node: optimizer/gradients/KernelPointNetwork/layer_0/resnetb_1/conv2/concat_1_grad/GatherV2_2/axis/_222 = _HostSendT=DT_INT32, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1469_...rV2_2/axis", _device="/job:localhost/replica:0/task:0/device:GPU:0"]]

Caused by op 'IteratorGetNext', defined at: File "training_S3DIS.py", line 213, in dataset.init_input_pipeline(config) File "/home/hwk/KPConv/datasets/common.py", line 749, in init_input_pipeline self.flat_inputs = iter.get_next() File "/home/hwk/anaconda3/envs/py3.5/lib/python3.5/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 410, in get_next name=name)), self._output_types, File "/home/hwk/anaconda3/envs/py3.5/lib/python3.5/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 2069, in iterator_get_next output_shapes=output_shapes, name=name) File "/home/hwk/anaconda3/envs/py3.5/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "/home/hwk/anaconda3/envs/py3.5/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py", line 454, in new_func return func(*args, **kwargs) File "/home/hwk/anaconda3/envs/py3.5/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3155, in create_op op_def=op_def) File "/home/hwk/anaconda3/envs/py3.5/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1717, in init self._traceback = tf_stack.extract_stack()

OutOfRangeError (see above for traceback): End of sequence [[Node: IteratorGetNext = IteratorGetNextoutput_shapes=[[?,3], [?,3], [?,3], [?,3], [?,3], ..., [?], [?,3], [?,3,3], [?], [?]], output_types=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_INT32, DT_FLOAT, DT_FLOAT, DT_INT32, DT_INT32], _device="/job:localhost/replica:0/task:0/device:CPU:0"]] [[Node: optimizer/gradients/KernelPointNetwork/layer_0/resnetb_1/conv2/concat_1_grad/GatherV2_2/axis/_222 = _HostSendT=DT_INT32, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1469_...rV2_2/axis", _device="/job:localhost/replica:0/task:0/device:GPU:0"]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "training_S3DIS.py", line 244, in trainer.train(model, dataset) File "/home/hwk/KPConv/utils/trainer.py", line 347, in train self.cloud_validation_error(model, dataset) File "/home/hwk/KPConv/utils/trainer.py", line 806, in cloud_validation_error preds = (sub_preds[dataset.validation_proj[i_val]]).astype(np.int32) IndexError: arrays used as indices must be of integer (or boolean) type

I am looking forward to your reply.

Hello.

I want to do semantic segmentation on unlabeled point cloud data (xyzrgb) using a pre-trained model. I will use a pre-trained model using S3DIS. What is input format, rgbxyz? I would like to know how to do semantic segmentation on my own data.

Thank you.

Dear @HuguesTHOMAS,

I was wondering if you were going to keep working on improving the model.

I think KPConv could be improved at least with 3 ways.

1. The convolution definition doesn't use the distance to the central point within its formula.

I could have been (a): g(yi) = sum (g(norm(yi) / R) * h(yi, xk) * Wk) or (b): g(yi) = sum (g(yi)* h(yi, xk) * Wk)) (a): g takes a value between 0 and 1 and performs a guidance on it. (b): g takes yi directly and does the same

It will be similar to this one, as an analogy for image convolution.

2. Sigma could be variable or the density of each point could be used as in

During training, I get mIoU=86% and mcIoU=85%, but it seems that you do not use all of data in test set. I use test_any_model.py to test the trained model at epoch 500, but I get poor performance (mIoU=40.7% and mcIoU=34.3%). How to fix this issue?

Hello, thank you for your sharing, I really appreciate your KPConv work and I am working on implement it in PyTorch. But I was confused when I try to understand the code of data preparation part because there are so many encapsulations or wrappers. So would you please give me some high level idea about the implementation. Basically my questions are

1. About the flat_inputs variable : In my understanding, you build the flat_inputs which consists of the points and its neighbors for each layer before session run. But how can you calculate the neighbors of each layer before the specific point cloud is sent into the network?
2. For the convolutional part, I already know how to build the matrix and calculate the output feature for the batch input, but before conv I need know the neighborhood indices of each point, I have no idea how to calculate the neighbor for each point cloud in parallel, obviously if I try to calculate the neighbor indices one by one and then concatenation it will be very time comsuing. So in your implementation, how do you solve this problem ?

What is the file format of the prediction results that need to be uploaded when testing the semantic3D dataset online?Can you show it in detail?The website description is very unclear. looking forward to your reply.

Thanks for your great work,after I have trained, I run python visualize_features.py,I meet this problem:

/KPConv-master2/utils/visualizer.py", line 220, in top_relu_activations l = np.argmax(np.bincount(labels[b])) IndexError: index 3 is out of bounds for axis 0 with size 3 can you help me?looking forward to your reply.

Thanks for your great work and nice code. I met some problems when I use the project on my own dataset,I have no idea how to deal with it. I hope you can help me. Caught a NaN error : 3 targets[872] is out of range [[node results/in_top_k/InTopKV2 (defined at /home/sanwei/下载/beifen/KPConvmy/utils/trainer.py:191) = InTopKV2[T=DT_INT32, _device="/job:localhost/replica/task:0/device:CPU:0"](KernelPointNetwork/softmax/GatherNd, IteratorGetNext:24, optimizer/gradients/range_9/delta)]] name: "results/in_top_k/InTopKV2" op: "InTopKV2" input: "KernelPointNetwork/softmax/GatherNd" input: "IteratorGetNext:24" input: "results/in_top_k/InTopKV2/k" attr { key: "T" value { type: DT_INT32 } }

results/in_top_k/InTopKV2 ['KernelPointNetwork/softmax/GatherNd:0', 'IteratorGetNext:24', 'results/in_top_k/InTopKV2/k:0'] ['results/in_top_k/InTopKV2:0'] Traceback (most recent call last): File "/home/sanwei/anaconda3/envs/KPconv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call return fn(*args) File "/home/sanwei/anaconda3/envs/KPconv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/home/sanwei/anaconda3/envs/KPconv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InvalidArgumentError: targets[872] is out of range [[{{node results/in_top_k/InTopKV2}} = InTopKV2[T=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"](KernelPointNetwork/softmax/GatherNd, IteratorGetNext:24, optimizer/gradients/range_9/delta)]]

Hello,

Thank you for sharing your KPCONV code! I'd like to ask if it is possible to add more features to the computation (i.e., XYZ, RGB, intensity or reflectance, no. of returns, return no.). I do not know if I am thinking right, but the changes should be don in datasets/Semantic3D.py (or other datasets.file) in line:

https://github.com/HuguesTHOMAS/KPConv/blob/16bfbb96ac9af48a3c829d1f8123152d35b63862/datasets/Semantic3D.py#L214-L236

by changing the structure of data file like XYZ, RGB, other and

        data = np.loadtxt(txt_file)


        points = data[:, :3].astype(np.float32)
        colors = data[:, 3:6].astype(np.uint8)
        **addvalues = data[:, 6:9].astype(np.float32)**

and

             sub_points, sub_colors, **sub_prop**, sub_labels = grid_subsampling(points,
                                                                      features=colors,
                                                                      **prop=addvalues,**
                                                                      labels=labels,
                                                                      sampleDl=0.01)
                # Write the subsampled ply file
                write_ply(ply_file_full, (sub_points, sub_colors, sub_prop, sub_labels), ['x', 'y', 'z', 'red', 'green', 'blue', **'reflectance', 'noreturn', 'returnno**', 'class'])
            else:

                # Write the full ply file
                write_ply(ply_file_full, (points, colors,**addvalues**), ['x', 'y', 'z', 'red', 'green', 'blue', **'reflectance', 'noreturn', 'returnno'**])

then in https://github.com/HuguesTHOMAS/KPConv/blob/16bfbb96ac9af48a3c829d1f8123152d35b63862/datasets/Semantic3D.py#L288-L315 should be 1 line added:

               data = read_ply(sub_ply_file)
                sub_colors = np.vstack((data['red'], data['green'], data['blue'])).T
                **sub_prop = np.vstack((data['reflectance'], data['noreturn'], data['returnno'])).T**

and in else also:

           else:

                # Read ply file
                data = read_ply(file_path)
                points = np.vstack((data['x'], data['y'], data['z'])).T
                colors = np.vstack((data['red'], data['green'], data['blue'])).T
                **sub_prop = np.vstack((data['reflectance'], data['noreturn'], data['returnno'])).T**
                if cloud_split == 'test':

and in

                # Subsample cloud
                sub_data = grid_subsampling(points,
                                            features=colors,
                                             **prop=addvalues,**
                                            labels=int_features,
                                            sampleDl=subsampling_parameter)

**But how about rescaling?

               # Rescale float color and squeeze label
                sub_prop = sub_data[2] / 50  #if the reflectance is used?

**

The next step is to change lines: https://github.com/HuguesTHOMAS/KPConv/blob/16bfbb96ac9af48a3c829d1f8123152d35b63862/datasets/Semantic3D.py#L327-L337

like:

                # Save ply
                if cloud_split == 'test':
                    sub_labels = None
                    write_ply(sub_ply_file,
                              [sub_data[0], sub_colors, sub_prop],
                              ['x', 'y', 'z', 'red', 'green', 'blue', **'reflectance', 'noreturn', 'returnno'**])
                else:
                    sub_labels = np.squeeze(sub_data[**3**])
                    write_ply(sub_ply_file,
                              [sub_data[0], sub_colors, sub_prop, sub_labels],
                              ['x', 'y', 'z', 'red', 'green', 'blue', **'reflectance', 'noreturn', 'returnno'**, 'class'])

and: https://github.com/HuguesTHOMAS/KPConv/blob/16bfbb96ac9af48a3c829d1f8123152d35b63862/datasets/Semantic3D.py#L339-L343

            # Fill data containers
            self.input_trees[cloud_split] += [search_tree]
            self.input_colors[cloud_split] += [sub_colors]
            **self.input_addvalues[cloud_split] +=[sub_prop]**
            if cloud_split in ['training', 'validation']:
                self.input_labels[cloud_split] += [sub_labels]

then in line: https://github.com/HuguesTHOMAS/KPConv/blob/16bfbb96ac9af48a3c829d1f8123152d35b63862/datasets/Semantic3D.py#L581-L585

                # Collect points and colors
                input_points = (points[input_inds] - pick_point).astype(np.float32)
                input_colors = self.input_colors[split][cloud_ind][input_inds]
                **input_addvalues=self.input_addvalues[split][cloud_ind][input_inds]**
                input_labels = self.input_labels[split][cloud_ind][input_inds]
                input_labels = np.array([self.label_to_idx[l] for l in input_labels])

in https://github.com/HuguesTHOMAS/KPConv/blob/16bfbb96ac9af48a3c829d1f8123152d35b63862/datasets/Semantic3D.py#L606 ** c_list += [np.hstack((input_colors, input_addvalues, input_points + pick_point))] **

in https://github.com/HuguesTHOMAS/KPConv/blob/16bfbb96ac9af48a3c829d1f8123152d35b63862/datasets/Semantic3D.py#L675-L677

                # Collect points and colors
                input_points = (points[input_inds] - pick_point).astype(np.float32)
                input_colors = self.input_colors[data_split][cloud_ind][input_inds]
                 **input_addvalues=self.input_addvalues[split][cloud_ind][input_inds]**

https://github.com/HuguesTHOMAS/KPConv/blob/16bfbb96ac9af48a3c829d1f8123152d35b63862/datasets/Semantic3D.py#L703 ** c_list += [np.hstack((input_colors, input_addvalues, input_points + pick_point))] **

Do the types and shapes have to be changed ? https://github.com/HuguesTHOMAS/KPConv/blob/16bfbb96ac9af48a3c829d1f8123152d35b63862/datasets/Semantic3D.py#L736-L738

and the last part: https://github.com/HuguesTHOMAS/KPConv/blob/16bfbb96ac9af48a3c829d1f8123152d35b63862/datasets/Semantic3D.py#L745-L792

        def tf_map(stacked_points, stacked_colors, stacked_addvalues, point_labels, stacks_lengths, point_inds, cloud_inds):


            # Get batch indice for each point
            batch_inds = self.tf_get_batch_inds(stacks_lengths)


            # Augment input points
            stacked_points, scales, rots = self.tf_augment_input(stacked_points,
                                                                 batch_inds,
                                                                 config)


            # First add a column of 1 as feature for the network to be able to learn 3D shapes
            stacked_features = tf.ones((tf.shape(stacked_points)[0], 1), dtype=tf.float32)


            # Get coordinates and colors
            stacked_original_coordinates = stacked_colors[:, 3:]
            stacked_colors = stacked_colors[:, :3]
**         # Get coordinates and addvalues
            stacked_original_coordinates = stacked_addvalues[:, 3:]
            stacked_addvalues = stacked_addvalues[:, :3]**

            # Augmentation : randomly drop colors
            if config.in_features_dim in [4, 5]:
                num_batches = batch_inds[-1] + 1
                s = tf.cast(tf.less(tf.random_uniform((num_batches,)), config.augment_color), tf.float32)
                stacked_s = tf.gather(s, batch_inds)
                stacked_colors = stacked_colors * tf.expand_dims(stacked_s, axis=1)


            # Then use positions or not
            if config.in_features_dim == 1:
                pass
            elif config.in_features_dim == 2:
                stacked_features = tf.concat((stacked_features, stacked_original_coordinates[:, 2:]), axis=1)
            elif config.in_features_dim == 3:
                stacked_features = stacked_colors
            elif config.in_features_dim == 4:
                stacked_features = tf.concat((stacked_features, stacked_colors), axis=1)
            elif config.in_features_dim == 5:
                stacked_features = tf.concat((stacked_features, stacked_colors, stacked_original_coordinates[:, 2:]), axis=1)
            elif config.in_features_dim == 7:
                stacked_features = tf.concat((stacked_features, stacked_colors, stacked_points), axis=1)
**            elif config.in_features_dim == 10:
                stacked_features = tf.concat((stacked_features, stacked_colors, stacked_addvalues, stacked_points), axis=1)**
            else:
                raise ValueError('Only accepted input dimensions are 1, 3, 4 and 7 , 10(without and with rgb/xyz)')

Is there something else should be changed or the changes should be done in different way?

I would like to test the KPConv on the pertrained model of Semntic3D dataset (any of semantic8 or reduced-8). Could you please share if you have the pretrained model?

Thanks

Hi Hugues, I am trying to train KPConv with the ModelNet40 dataset, but after the message "Dataset Preparation

Loading training points", it gives me the message below and finishes the whole process without giving any error. Do you have any idea why this might have happened?