Hello, I am very interested in your work. Thank you for sharing such an excellent article. I have a question about the config of crowdpose. Why ROOT: 'data/coco/', TEST_SET: 'val2017' TRAIN_SET: 'train2017'?
TEST: BATCH_SIZE_PER_GPU: 24

TRAIN_SET: 'train2017'

i think 'data/coco/' is 'data/crowdpose/'; TEST_SET: 'crowdpose_val2017' TRAIN_SET: 'crowdpose_train2017' ，Don't know if my understanding is correct

Hi, I try to training my coco style dataset by your scripts, I dont know which bash script should be used to train.(Could you please briefly explain the function of each script?) Then I use "scripts/train/lambda/coco/train.sh" this one for training. but one error happened.

cd /data_2/lutianhao/code/MIPNet/ CUDA_VISIBLE_DEVICES=4,5,6,7, python tools/lambda/train_lambda_real.py
--cfg experiments/coco/hrnet/w48_384x288_adam_lr1e-3.yaml
GPUS '(0,1,2,3,)'
OUTPUT_DIR 'Outputs/outputs/lambda/lambda_coco_real_waffle'
LOG_DIR 'Outputs/logs/lambda/lambda_coco_real_waffle'
TEST.MODEL_FILE 'models/pytorch/pose_coco/pose_hrnet_w48_384x288.pth'
DATASET.TRAIN_DATASET 'coco_lambda'
DATASET.TRAIN_SET 'train2017'
DATASET.TRAIN_IMAGE_DIR '/data_2/lutianhao/datasets/pose/coco2017/train2017'
DATASET.TRAIN_ANNOTATION_FILE '/data_2/lutianhao/datasets/pose/coco2017/annotations/person_keypoints_train2017.json'
DATASET.TRAIN_DATASET_TYPE 'coco_lambda'
DATASET.TEST_DATASET 'coco'
DATASET.TEST_SET 'val2017'
DATASET.TEST_IMAGE_DIR '/data_2/lutianhao/datasets/pose/coco2017/val2017'
DATASET.TEST_ANNOTATION_FILE '/data_2/lutianhao/datasets/pose/coco2017/annotations/person_keypoints_val2017.json'
DATASET.TEST_DATASET_TYPE 'coco'
TRAIN.LR 0.001
TRAIN.BEGIN_EPOCH 0
TRAIN.END_EPOCH 110
TRAIN.LR_STEP '(70, 100)'
TRAIN.BATCH_SIZE_PER_GPU 2
TEST.BATCH_SIZE_PER_GPU 1
TEST.USE_GT_BBOX True
EPOCH_EVAL_FREQ 1
PRINT_FREQ 100
MODEL.NAME 'pose_hrnet_se_lambda'
MODEL.SE_MODULES '[False, False, True, True]'

And the error is :

GAMMA1: 0.99 [0/927] GAMMA2: 0.0 LR: 0.001 LR_FACTOR: 0.1 LR_STEP: [70, 100] MOMENTUM: 0.9 NESTEROV: False OPTIMIZER: adam RESUME: False SHUFFLE: True WD: 0.0001 WORKERS: 24 => init weights from normal distribution => loading pretrained model models/pytorch/imagenet/hrnet_w48-8ef0771d.pth

Total Parameters: 63,746,081

Total Multiply Adds (For Convolution and Linear Layers only): 46.562052726745605 GFLOPs

Number of Layers Conv2d : 293 layers BatchNorm2d : 292 layers ReLU : 271 layers Bottleneck : 4 layers BasicBlock : 104 layers Upsample : 28 layers HighResolutionModule : 8 layers AdaptiveAvgPool2d : 5 l ayers Linear : 20 layers Sigmoid : 10 layers BatchNorm1d : 5 layers SELambdaLayer : 5 layers SELambdaModule : 2 layers => loading model from models/pytorch/pose_coco/pose_hrnet_w48_384x288.pth => loading from latest_state_dict at models/pytorch/pose_coco/pose_hrnet_w48_384x288.pth loading annotations into memory... Done (t=31.87s) creating index... index created! => classes: ['background', 'person'] => num_images: 118287 loading from cache from cache/coco_lambda/train2017/gt_db.pkl done! => load 149813 samples loading annotations into memory... Done (t=4.04s) creating index... index created! => classes: ['background', 'person'] => num_images: 5000 => load 6352 samples => resuming optimizer from models/pytorch/pose_coco/pose_hrnet_w48_384x288.pth => updated lr schedule is [70, 100]

training on lambda Epoch: [0][0/18727] Time 64.338s (64.338s) Speed 0.2 samples/s Data 10.114s (10.114s) Loss 0.00020 (0.00020) Accuracy 0.513 (0.513) model_grad 0.000568 (0.000568) DivLoss -0.00074 (-0.00074) PoseLoss 0.00020 (0.00020) Traceback (most recent call last): File "tools/lambda/train_lambda_real.py", line 280, in main() File "tools/lambda/train_lambda_real.py", line 242, in main final_output_dir, tb_log_dir, writer_dict, print_prefix='lambda') File "/data_2/lutianhao/code/MIPNet/tools/lambda/../../lib/core/train.py", line 464, in train_lambda suffix += '_[{}:{}]'.format(count, round(lambda_a[count + B].item(), 2)) IndexError: index 16 is out of bounds for dimension 0 with size 16

Amazing work! Thanks for releasing the source code. Here is my question:

1. Which of the following is MIPNet(hrnet backbone)'s final result on crowdpose test set in the paper? #----------------------------------------------------------------------------------- [code] lib/core/validate.py: def validate_lambda_quantitative(): _print_name_value(name_values, 'l0,1:{}'.format(model_name)) _print_name_value(name_values_mode0, 'l0:{}'.format(model_name)) _print_name_value(name_values_mode1, 'l1:{}'.format(model_name)) #------------------------------------------------------------------------------------
2. Has MIPNet with hrnet backbone used pretrained model? (about result on crowdpose in the paper)

By the way, Very looking forward to the detailed README and final checkpoints trained on dataset.

Hello,just checked out your paper.Great work! I had the following questions - 1)Is there a approximate deadline for the code with instructions to be released for training and testing the model for our custom use-case? 2)The annotation for the ground truth generation is just the 2d key points of the particular object right,the network will automatically convert it to heatmap or do we have to give it as a heatmap? 3)Can we apply this to a case as follows- A camera is observing 2-3 moving robots in its FOV,and if I retrain the network with the images of the robot,with its corresponding ground-truth keypoints,the network can still predict the 2d keypoints of the moving robot right? 4)For this does the robot have to be at a particular distance from the camera,so that the keypoint estimation is accurate enough?Meaning is the network's accuracy dependent on "the distance the object is from the camera"?

Any suggestions/replies are greatly appreciated! Thank you

Hi, thank you for your nice work here. In your paper, in section 4.1, it is said that the detection bbox is extended to a fixed aspect ratio of 4:3, then this bbox is cropped from the original full image. I'm having troubles trying to find the part of code where these operations are done. Could you point me to it? Thanks!

Hello, your work is very good, but when I tested OChuman's dataset, I found that the path in the shell file is ocpose, and what I want to teach is: I changed the ocpose to ochuman according to my ideas, in the training, this problem arose, The train_loader is missing, can you help me to solve my issues. thanks!

=> classes: ['background', 'person'] => num_images: 2231 => load 3819 samples => updated lr schedule is [170, 200] training on lambda Traceback (most recent call last): File "tools/lambda/train_lambda_real.py", line 280, in main() File "tools/lambda/train_lambda_real.py", line 241, in main train_lambda(cfg, train_loader, model, criterion_lambda, criterion, optimizer, epoch, File "/Workspace/MIPNet-main/MIPNet-main/tools/lambda/../../lib/core/train.py", line 355, in train_lambda for i, (input, target_a, target_weight_a, meta_a, target_b, target_weight_b, meta_b) in enumerate(train_loader): ValueError: not enough values to unpack (expected 7, got 4)

Hi, there might be a possible bug in train_lambda_real.py (L249 - L225):

In each epoch, the perf_indicator is directly set to 0.0. In other words, the new best model is achieved in each epoch and the model in the last epoch will be the final best model. However, this is likely not correct, as the best model may not always be achieved in the last epoch.

In my opinion, the perf_indicator should be the return value of validate_lambda_quantitative() in L245.

Amazing work and thanks for sharing the code!

I have a question regarding training: in scripts/train there're two folders named "lambda" and "lambda_general". What is the difference between them? Which one should I use, if I want to train and reproduce the results in the paper?

Thanks in advance!

Best Regards