Grateful for your implementation! I have some questions about the bilinear sampler.

https://github.com/octiapp/KerasPersonLab/blob/32d44dd1f33377128a87d6e074cf8214224f0174/model.py#L49 Why base = base + bilinear_sampler(offsets, base) instead of base = base + bilinear_sampler(base, offsets) ? Does this mean that we interpolate the offset according to the base ?

https://github.com/octiapp/KerasPersonLab/blob/32d44dd1f33377128a87d6e074cf8214224f0174/bilinear.py#L43 If we iy0 = vy0 + h, then the iy0 will represent the destination position of offsets. Why using these positions when iy0 = tf.where(mask, tf.zeros_like(iy0), iy0) ?

In this condition,

x00 = tf.gather_nd(x, i00)
x01 = tf.gather_nd(x, i01)
x10 = tf.gather_nd(x, i10)
x11 = tf.gather_nd(x, i11)

will gather values from the destination positions of predicted offsets, shouldn't we gather values from the start positions of offsets?

hi，all One error happens when training from scratch.

KerasPersonLab-updated_model_def/data_prep.py", line 16, in get_ground_truth

assert(instance_masks.shape[-1] == len(all_keypoints)), '{} != {}'.format(instance_masks.shape[-1], len(all_keypoints))

AssertionError: 7 != 8

10993/16028 [===================>..........] - ETA: 59:31 - loss: 152848912.0000Traceback (most recent c all last): File "train.py", line 77, in epochs=config.NUM_EPOCHS, callbacks=callbacks) File "/root/anaconda3/envs/deep_source/lib/python3.6/site-packages/keras/engine/training.py", line 103 7, in fit validation_steps=validation_steps) File "/root/anaconda3/envs/deep_source/lib/python3.6/site-packages/keras/engine/training_arrays.py", l ine 154, in fit_loop outs = f(ins) File "/root/anaconda3/envs/deep_source/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py ", line 2666, in call return self._call(inputs) File "/root/anaconda3/envs/deep_source/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2636, in _call fetched = self._callable_fn(*array_vals) File "/root/anaconda3/envs/deep_source/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1382, in call run_metadata_ptr) File "/root/anaconda3/envs/deep_source/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 519, in exit c_api.TF_GetCode(self.status.status)) tensorflow.python.framework.errors_impl.UnknownError: AssertionError: Traceback (most recent call last):

File "/root/anaconda3/envs/deep_source/lib/python3.6/site-packages/tensorflow/python/ops/script_ops.py", line 206, in call ret = func(*args)

File "/root/networks/KerasPersonLab/tf_data_generator.py", line 71, in transform_data kp_maps, short_offsets, mid_offsets, long_offsets = get_ground_truth(instance_masks, kp)

File "/root/networks/KerasPersonLab/data_prep.py", line 16, in get_ground_truth assert(instance_masks.shape[-1] == len(all_keypoints))

AssertionError

The error pretty much explain itself, not sure why shape is mis-matching.

Hi!

First, thank you for your hard work!

I am trying to run training and some time after the first epoch begins, I am running into an issue where the following call (in transformer.py) fails:

masks = cv2.warpAffine(masks, M, cv_shape, flags=cv2.INTER_CUBIC, borderMode=cv2.BORDER_CONSTANT, borderValue=0)

Specifically, there is an OpenCV assertion error, stating that "error: (-215) _src.channels() <= 4 || (interpolation != INTER_LANCZOS4 && interpolation != INTER_CUBIC) in function cv::warpAffine". Could you please point me at what might be wrong here?

Thanks a lot!

N.B. Looking at the code, at first glace it would so seem that several Mats are passed into warpAffine(), is this intentional?

It seems like that no dilation convolution in origin PersonLab paper, according to classical Resnet . Did I miss or ignore something important in paper?

https://github.com/octiapp/KerasPersonLab/blob/32d44dd1f33377128a87d6e074cf8214224f0174/post_proc.py#L97

From my understanding,

this_skel[kp['id'], :2] = kp['xy']
this_skel[kp['id'], 2] = kp['conf']
path = iterative_bfs(skeleton_graph, kp['id'])[1:]
for edge in path:
      if this_skel[edge[0],2] == 0:
           continue

the order of the path will be uniquely determined by the start keypoint due to the tree-structure of person. So, if some keypoints in the path are not detected, multiple skeletons will be generated for a single body because of this snippet:

if this_skel[edge[0],2] == 0:
           continue

for example, let's assume that the start (the highest score) keypoint is Lwrist and all keypoints except L_elbow are detected for a single person, then this for edge in path: loop will always goto continue and this_skeleton will only contain the Lwrist keypoint location.

I don't know if I understand it correctly?

when I use python 3, I get the following Error: TypeError: 'KeysView' object does not support indexing so when use python 3, I should modify the function data_gen() in tf_data_generator.py in line87: key = root.keys() to key = list(root.keys()) and now key could be iterated. I think you should tell the environment detail in README so that other people could avoid such mistakes

When running the following code:

parallel_model.fit(steps_per_epoch=64115//batch_size, 
                   epochs=config.NUM_EPOCHS, 
                   callbacks=callbacks)

Getting this error.

WARNING:tensorflow:From /usr/local/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
Epoch 1/125
---------------------------------------------------------------------------
UnknownError                              Traceback (most recent call last)
<ipython-input-11-a6811e3578ae> in <module>
      1 parallel_model.fit(steps_per_epoch=64115//batch_size, 
      2                    epochs=config.NUM_EPOCHS,
----> 3                    callbacks=callbacks)

/usr/local/lib/python3.7/site-packages/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, **kwargs)
   1037                                         initial_epoch=initial_epoch,
   1038                                         steps_per_epoch=steps_per_epoch,
-> 1039                                         validation_steps=validation_steps)
   1040 
   1041     def evaluate(self, x=None, y=None,

/usr/local/lib/python3.7/site-packages/keras/engine/training_arrays.py in fit_loop(model, f, ins, out_labels, batch_size, epochs, verbose, callbacks, val_f, val_ins, shuffle, callback_metrics, initial_epoch, steps_per_epoch, validation_steps)
    152                 batch_logs['size'] = 1
    153                 callbacks.on_batch_begin(step_index, batch_logs)
--> 154                 outs = f(ins)
    155 
    156                 outs = to_list(outs)

/usr/local/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py in __call__(self, inputs)
   2713                 return self._legacy_call(inputs)
   2714 
-> 2715             return self._call(inputs)
   2716         else:
   2717             if py_any(is_tensor(x) for x in inputs):

/usr/local/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py in _call(self, inputs)
   2673             fetched = self._callable_fn(*array_vals, run_metadata=self.run_metadata)
   2674         else:
-> 2675             fetched = self._callable_fn(*array_vals)
   2676         return fetched[:len(self.outputs)]
   2677 

/usr/local/lib/python3.7/site-packages/tensorflow/python/client/session.py in __call__(self, *args, **kwargs)
   1437           ret = tf_session.TF_SessionRunCallable(
   1438               self._session._session, self._handle, args, status,
-> 1439               run_metadata_ptr)
   1440         if run_metadata:
   1441           proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

/usr/local/lib/python3.7/site-packages/tensorflow/python/framework/errors_impl.py in __exit__(self, type_arg, value_arg, traceback_arg)
    526             None, None,
    527             compat.as_text(c_api.TF_Message(self.status.status)),
--> 528             c_api.TF_GetCode(self.status.status))
    529     # Delete the underlying status object from memory otherwise it stays alive
    530     # as there is a reference to status from this from the traceback due to

UnknownError: OSError: Unable to open file (unable to lock file, errno = 35, error message = 'Resource temporarily unavailable')
Traceback (most recent call last):

  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/ops/script_ops.py", line 207, in __call__
    ret = func(*args)

  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 449, in generator_py_func
    values = next(generator_state.get_iterator(iterator_id))

  File "/Users/ghafran/git/KerasPersonLab/tf_data_generator.py", line 85, in data_gen
    h5 = h5py.File(config.H5_DATASET, 'r')

  File "/usr/local/lib/python3.7/site-packages/h5py/_hl/files.py", line 394, in __init__
    swmr=swmr)

  File "/usr/local/lib/python3.7/site-packages/h5py/_hl/files.py", line 170, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)

  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper

  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper

  File "h5py/h5f.pyx", line 85, in h5py.h5f.open

OSError: Unable to open file (unable to lock file, errno = 35, error message = 'Resource temporarily unavailable')


	 [[{{node PyFunc}}]]
	 [[{{node IteratorGetNext}}]]

Hi, thanks a lot for the implementation. there's something that i couldn't figure out after reading the paper and reviewing the code. during the computation of the heatmaps the accumulate_votes function is called and some sort of bilinear interpolation is made. could someone please clarify which function is interpolated (in the code it seems to me that the values of this function are stored in the variable ps) and where do the interpolation weights come from (dx,dy) and what do they mean? i'm referring to the following part of the code in the post_proc.py: def accumulate_votes(votes, shape): xs = votes[:,0] ys = votes[:,1] ps = votes[:,2] tl = [np.floor(ys).astype('int32'), np.floor(xs).astype('int32')] tr = [np.floor(ys).astype('int32'), np.ceil(xs).astype('int32')] bl = [np.ceil(ys).astype('int32'), np.floor(xs).astype('int32')] br = [np.ceil(ys).astype('int32'), np.ceil(xs).astype('int32')] dx = xs - tl[1] dy = ys - tl[0] tl_vals = ps*(1.-dx)(1.-dy) tr_vals = psdx*(1.-dy) bl_vals = psdy(1.-dx) br_vals = psdydx

Does someone have managed to get results with this repository?

My loss is near constant, oscillating around 1.50. I should mention that I have removed the instance segmentation module, but that shouldn't impact the pose estimation module.

Thank you for your reproducing of PersonLab. I used it to train model on COCO train2017 and got a not bad intermediate result. However, when I use the model to detect a 1K(1920x1080) image with about 40 persons, the Inference Time is very long(about 3s do all detections, and 13s do group matching) even with GPUs. I found it mainly caused by the stage of Group Joints, especially the function compute_heatmaps() with inputs kp_maps, short_offsets which is the refinement of Keypoints Heatmaps. Theoretically, Inference Time of bottom-up method will not grow linearly when persons increase. Is there a problem with the implementation of the function compute_heatmaps()?

Hi @jricheimer

Thank you so much for your implementation, It would be great if we can get access to pretrained model to compare it's accuracy with paper.

Thanks,

I've encountered a couple of errors in my code I was hoping you could help me fix:

1. I used get_resnet50_base and when i tried to load the weights this error popped up ValueError: You are trying to load a weight file containing 111 layers into a model with 116 layers.
2. if I put get_resnet50_base as my build base, which intermidiate layer do I use?