Hi, outputs/segms/train/Capture9/0129_indextip/cam410062/image32412.png. This segmentation has a value of '33', which is not expected. Do you have any ideas?

Hi,

I read the paper and it looks interesting. I was wondering how you would run this model on a single image. The provided demo code seems to only run on the validation set.

So if I have an image for which I have the bounding box/segmentation data of the hands, then how should I preprocess this for the model?

Given that this code is based on InterHand2.6M I tried to do something similar to what they have in their demo code:

def demo_model(args):
    pl.seed_everything(args.seed)
    model = fetch_pl_model(args, args.experiment)

    model.cuda()
    model.freeze()
    model.eval()

    # prepare input image
    transform = transforms.ToTensor()
    img_path = '1.jpg'
    original_img = cv2.imread(img_path)
    original_img_height, original_img_width = original_img.shape[:2]

    # prepare bbox
    bbox = [723, 354, 127, 74] # xmin, ymin, width, height  1
    bbox = process_bbox(bbox, (original_img_height, original_img_width, original_img_height))
    img, trans, inv_trans = generate_patch_image(original_img, bbox, False, 1.0, 0.0, cfg.input_img_shape)
    img = transform(img.astype(np.float32))/255
    img = img.cuda()[None,:,:,:]

    # forward
    inputs = {'img': img}
    targets = {}
    meta_info = {}

    with torch.no_grad():
        vis_dict = model(inputs, targets, meta_info, 'vis')
        print("vis_dict", vis_dict)`

Where process_bbox and generate_patch_image are from InterHand2.6M however, it didn't work out for me. The model always returned None. My guess is that it requires the bbox to be processed differently, looking at the code I think it has to do something with the "segm_256" key in the "targets" dict, but not sure what would be the easiest way.

I would appreciate some help or directions on what would be the simplest way to write a script that runs the model on a single image.

Best regards, Kevin

Hi, I ran into a problem while following your README to run the code. I wonder whether it's because I ran it on CUDA11.3 did I run the render_mano_ih operation incorrectly, and there is a problem with the lmdb file obtained..

When I train DIGIT or test it, I met the problem with dataloader：

Number of annotations in single hand sequences: 58463 Number of annotations in interacting hand sequences: 36568 Total number of annotations: 95031

Defrost ALL GPU available: True, used: True TPU available: False, using: 0 TPU cores CUDA_VISIBLE_DEVICES: [0] Using native 16bit precision.

| Name | Type | Params

0 | model | Model | 40 M

Validation sanity check: 0it [00:00, ?it/s]ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm). ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm). ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm). ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm). ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm). ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm). ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm). ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm). Traceback (most recent call last): File "/home/zhenghanmo/anaconda3/envs/digit/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 872, in _try_get_data data = self._data_queue.get(timeout=timeout) File "/home/zhenghanmo/anaconda3/envs/digit/lib/python3.7/queue.py", line 179, in get self.not_empty.wait(remaining) File "/home/zhenghanmo/anaconda3/envs/digit/lib/python3.7/threading.py", line 300, in wait gotit = waiter.acquire(True, timeout) File "/home/zhenghanmo/anaconda3/envs/digit/lib/python3.7/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler _error_if_any_worker_fails() RuntimeError: DataLoader worker (pid 4844) is killed by signal: Bus error. It is possible that dataloader's workers are out of shared memory. Please try to raise your shared memory limit.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "train.py", line 56, in run_exp(cfg) File "train.py", line 51, in run_exp trainer.fit(model, train_loader, [val_loader]) File "/home/zhenghanmo/anaconda3/envs/digit/lib/python3.7/site-packages/pytorch_lightning/trainer/states.py", line 48, in wrapped_fn result = fn(self, *args, **kwargs) File "/home/zhenghanmo/anaconda3/envs/digit/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1073, in fit results = self.accelerator_backend.train(model) File "/home/zhenghanmo/anaconda3/envs/digit/lib/python3.7/site-packages/pytorch_lightning/accelerators/gpu_backend.py", line 51, in train results = self.trainer.run_pretrain_routine(model) File "/home/zhenghanmo/anaconda3/envs/digit/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1224, in run_pretrain_routine self._run_sanity_check(ref_model, model) File "/home/zhenghanmo/anaconda3/envs/digit/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1257, in _run_sanity_check eval_results = self._evaluate(model, self.val_dataloaders, max_batches, False) File "/home/zhenghanmo/anaconda3/envs/digit/lib/python3.7/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 305, in _evaluate for batch_idx, batch in enumerate(dataloader): File "/home/zhenghanmo/anaconda3/envs/digit/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 435, in next data = self._next_data() File "/home/zhenghanmo/anaconda3/envs/digit/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1068, in _next_data idx, data = self._get_data() File "/home/zhenghanmo/anaconda3/envs/digit/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1024, in _get_data success, data = self._try_get_data() File "/home/zhenghanmo/anaconda3/envs/digit/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 885, in _try_get_data raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e RuntimeError: DataLoader worker (pid(s) 4844) exited unexpectedly.

However, when I change num_workers to 0, I met another problem：

Using native 16bit precision.

| Name | Type | Params

0 | model | Model | 40 M

/home/zhenghanmo/anaconda3/envs/digit/lib/python3.7/site-packages/pytorch_lightning/utilities/distributed.py:37: UserWarning: The dataloader, val dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the num_workers argument(try 36 which is the number of cpus on this machine) in theDataLoader` init to improve performance. warnings.warn(*args, **kwargs) Validation sanity check: 0it [00:00, ?it/s]Bus error (core dumped)

Hi,

Q1. I was trying to understand this part and Fig 6c. of your paper. What is segmentation probability distribution? Do you mean (B, class_num, H, W) before doing argmax into (B, 1, H, W)

You mentioned you trained two different pipelines using Fig6c. But I couldn't find a correspondence in Fig6c. Do you mean that you replace Segm.(S) with (B, class_num, H, W)and (B, 1, H, W) separately?

Q2. What is the corresponding class of the part-segmentation label ? is it the same as original MANO's joints without tips or InterHands2.6M's order? (i.e. wrist is 1 or 16?)

Q3. https://github.com/zc-alexfan/digit-interacting/blob/36e5110ac91b0e6ac2151d43d5b1cc5712037080/src/dataset/dataset_utils.py#L133-L149 What is img_segm_swapped[1], img_segm_swapped[2] = img_segm_swapped[2].clone(), img_segm_swapped[1].clone() used for? I think this operation is conducted on the channel dimension, I'm not sure what is this for. The 3 channels should be the same.

Q4. Why don't you ignore background label when training the segmentation?

Thank you for your excellent research.

I wonder the meaning of the code package_images_lmdb.py in scripts.

1. At line 12, why fnames is redefined as fnames[:1000]?
2. At line 14, why 5130240 is multiplied to len(fnames)?

I'll be waiting for the reply. Thank you.

Hi,

Thanks for your great work, I just want to confirm that what data split you are using in table 1 and table 2 separately, because they have different results. Your MPJPE is 16.55 in table 2 but 14.27 in table 1