I got 19.27 for moderate AP3D with IOU threshold 0.7 on car category on PVRCNN with ROS from Waymo->KITTI, which is worse than the performance of that on PVRCNN without ROS shown in the paper(22.01). Then I trained the PVRCNN without ROS on my machine, and got the best result over 22 on epoch. I used the original config on cfgs/da-waymo-kitti_models/pvrcnn/ pvrcnn_old_anchor.yaml and cfgs/da-waymo-kitti_models/pvrcnn/ pvrcnn_old_anchor_ros.yaml, did I do something wrong? And did you also get the unstable result on Waymo->KITTI?

I am getting an error when running train.py, it seems to have something to do with PSEUDO_LABEL not being updated. The Traceback repeats for multiple frames, not just 002080 as seen below. I've also put the full output on this gist, in case the information below is not enough. Am I missing something? Thanks for any help!

Commands Run

$ NUM_GPUS=8
$ CONFIG_FILE=cfgs/da-waymo-kitti_models/pvrcnn_st3d/pvrcnn_st3d.yaml
$ bash scripts/dist_train.sh ${NUM_GPUS} --cfg_file ${CONFIG_FILE}

Error

[2021-07-21 15:05:09,022  train.py 168  INFO]  **********************Start training da-waymo-kitti_models/pvrcnn_st3d/pvrcnn_st3d(default)**********************
generate_ps_e0: 100%|████████████████████| 232/232 [03:14<00:00,  1.19it/s, pos_ps_box=0.000(0.000), ign_ps_box=15.000(14.899)]
Traceback (most recent call last):                                                                                             
  File "train.py", line 199, in <module>
    main()
  File "train.py", line 191, in main
    ema_model=None
  File "/home/user5/open-mmlab/ST3D/tools/train_utils/train_st_utils.py", line 157, in train_model_st
    dataloader_iter=dataloader_iter, ema_model=ema_model
  File "/home/user5/open-mmlab/ST3D/tools/train_utils/train_st_utils.py", line 42, in train_one_epoch_st
    target_batch = next(dataloader_iter)
  File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 582, in __next__
    return self._process_next_batch(batch)
  File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 608, in _process_next_batch
    raise batch.exc_type(batch.exc_msg)
ValueError: Traceback (most recent call last):
  File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 99, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 99, in <listcomp>
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/home/user5/open-mmlab/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 413, in __getitem__
    self.fill_pseudo_labels(input_dict)
  File "/home/user5/open-mmlab/ST3D/tools/../pcdet/datasets/dataset.py", line 146, in fill_pseudo_labels
    gt_boxes = self_training_utils.load_ps_label(input_dict['frame_id'])
  File "/home/user5/open-mmlab/ST3D/tools/../pcdet/utils/self_training_utils.py", line 221, in load_ps_label
    raise ValueError('Cannot find pseudo label for frame: %s' % frame_id)
ValueError: Cannot find pseudo label for frame: 002080

epochs:   0%|                                                                                           | 0/30 [04:05<?, ?it/s]

Environment

Python 3.7 CUDA 10.0 PyTorch 1.1 spconv 1.0 (commit 8da6f96) pcdet 0.2.0+73dda8c

Hi，jihan, the parameter SHIFT_COOR in cfg file pvrcnn_old_anchor_ros.yaml means what? does SHIFT_COOR is the lidar position difference among different dataset?

Hi I was just wondering how many epochs did you train on the nuscenes - kitti baseline (without SN or ROS) to get the result of 17.92 for 3D AP as reported in the paper? I've trained the secondiou_old_anchor.yaml cfg file for 3 epochs (as with ROS and SN) and only got the following results:

Car [email protected], 0.70, 0.70: bbox AP:60.1853, 48.0934, 48.1308 bev AP:31.2696, 27.1278, 26.6632 3d AP:4.5068, 3.2667, 3.3441 aos AP:42.78, 34.73, 35.04

Infos were generated with the original OpenPCDet repo - do I need to regenerate them for KITTI and nuscenes?

Also I noticed when evaluating the nuscenes-kitti for SN and ROS, they were only trained to 3 epochs. I re-trained the nuscenes dataset with the exact same secondiou_old_anchor_ros.yaml and didn't manage to reproduce the result from the model zoo. Update: I've re-generated the infos with st3d repo and it gives a lower performance for secondiou_old_anchor_ros.yaml.

Training was done with the command

python train.py --cfg_file cfgs/da-nuscenes-kitti_models/secondiou/secondiou_old_anchor_ros.yaml --batch_size 4 --epochs 3 --extra_tag st3d_infos

Cfg file for secondiou_old_anchor_ros.yaml is as below. Dataset_config files were unchanged.

CLASS_NAMES: ['car']

DATA_CONFIG:
    _BASE_CONFIG_: cfgs/dataset_configs/da_nuscenes_kitti_dataset.yaml
    MAX_SWEEPS: 1
    PRED_VELOCITY: False
    BALANCED_RESAMPLING: False
    SHIFT_COOR: [0.0, 0.0, 1.8]
    
    DATA_AUGMENTOR:
        DISABLE_AUG_LIST: ['normalize_object_size']
        AUG_CONFIG_LIST:
            - NAME: random_object_scaling
              SCALE_UNIFORM_NOISE: [0.75, 1.0]

            - NAME: normalize_object_size
              SIZE_RES: [-0.75, -0.34, -0.2]

            - NAME: random_world_flip
              ALONG_AXIS_LIST: ['x', 'y']

            - NAME: random_world_rotation
              WORLD_ROT_ANGLE: [-0.3925, 0.3925]

            - NAME: random_world_scaling
              WORLD_SCALE_RANGE: [0.95, 1.05]

DATA_CONFIG_TAR:
    _BASE_CONFIG_: cfgs/dataset_configs/da_kitti_dataset.yaml
    TARGET: True
    FOV_POINTS_ONLY: False
    CLASS_NAMES: ['Car']
    SHIFT_COOR: [0.0, 0.0, 1.6]

MODEL:
    NAME: SECONDNetIoU

    VFE:
        NAME: MeanVFE

    BACKBONE_3D:
        NAME: VoxelBackBone8x

    MAP_TO_BEV:
        NAME: HeightCompression
        NUM_BEV_FEATURES: 256

    BACKBONE_2D:
        NAME: BaseBEVBackbone

        LAYER_NUMS: [5, 5]
        LAYER_STRIDES: [1, 2]
        NUM_FILTERS: [128, 256]
        UPSAMPLE_STRIDES: [1, 2]
        NUM_UPSAMPLE_FILTERS: [256, 256]

    DENSE_HEAD:
        NAME: AnchorHeadSingle
        CLASS_AGNOSTIC: False

        USE_DIRECTION_CLASSIFIER: True
        DIR_OFFSET: 0.78539
        DIR_LIMIT_OFFSET: 0.0
        NUM_DIR_BINS: 2

        ANCHOR_GENERATOR_CONFIG: [
            {
                'class_name': 'car',
                'anchor_sizes': [[4.2, 2.0, 1.6]],
                'anchor_rotations': [0, 1.57],
                'anchor_bottom_heights': [0],
                'align_center': False,
                'feature_map_stride': 8,
                'matched_threshold': 0.55,
                'unmatched_threshold': 0.4
            }
        ]

        TARGET_ASSIGNER_CONFIG:
            NAME: AxisAlignedTargetAssigner
            POS_FRACTION: -1.0
            SAMPLE_SIZE: 512
            NORM_BY_NUM_EXAMPLES: False
            MATCH_HEIGHT: False
            BOX_CODER: ResidualCoder

        LOSS_CONFIG:
            LOSS_WEIGHTS: {
                'cls_weight': 1.0,
                'loc_weight': 2.0,
                'dir_weight': 0.2,
                'code_weights': [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
            }

    ROI_HEAD:
        NAME: SECONDHead
        CLASS_AGNOSTIC: True

        SHARED_FC: [256, 256]
        IOU_FC: [256, 256]
        DP_RATIO: 0.3

        NMS_CONFIG:
            TRAIN:
                NMS_TYPE: nms_gpu
                MULTI_CLASSES_NMS: False
                NMS_PRE_MAXSIZE: 9000
                NMS_POST_MAXSIZE: 512
                NMS_THRESH: 0.8
            TEST:
                NMS_TYPE: nms_gpu
                MULTI_CLASSES_NMS: False
                NMS_PRE_MAXSIZE: 1024
                NMS_POST_MAXSIZE: 100
                NMS_THRESH: 0.7

        ROI_GRID_POOL:
            GRID_SIZE: 7
            IN_CHANNEL: 512
            DOWNSAMPLE_RATIO: 8

        TARGET_CONFIG:
            BOX_CODER: ResidualCoder
            ROI_PER_IMAGE: 128
            FG_RATIO: 0.5

            SAMPLE_ROI_BY_EACH_CLASS: True
            CLS_SCORE_TYPE: raw_roi_iou

            CLS_FG_THRESH: 0.75
            CLS_BG_THRESH: 0.25
            CLS_BG_THRESH_LO: 0.1
            HARD_BG_RATIO: 0.8

            REG_FG_THRESH: 0.55

        LOSS_CONFIG:
            IOU_LOSS: BinaryCrossEntropy
            LOSS_WEIGHTS: {
                'rcnn_iou_weight': 1.0,
                'code_weights': [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
            }

    POST_PROCESSING:
        RECALL_THRESH_LIST: [0.3, 0.5, 0.7]
        SCORE_THRESH: 0.1
        OUTPUT_RAW_SCORE: False

        EVAL_METRIC: kitti

        NMS_CONFIG:
            MULTI_CLASSES_NMS: False
            NMS_TYPE: nms_gpu
            NMS_THRESH: 0.01
            NMS_PRE_MAXSIZE: 4096
            NMS_POST_MAXSIZE: 500


OPTIMIZATION:
    OPTIMIZER: adam_onecycle
    LR: 0.003
    WEIGHT_DECAY: 0.01
    MOMENTUM: 0.9

    MOMS: [0.95, 0.85]
    PCT_START: 0.4
    DIV_FACTOR: 10
    DECAY_STEP_LIST: [35, 45]
    LR_DECAY: 0.1
    LR_CLIP: 0.0000001

    LR_WARMUP: False
    WARMUP_EPOCH: 1

    GRAD_NORM_CLIP: 10
``

hi, sorry for bother you, i cannot reproduce your results of nuscene->kitti in v0.3.0. i have noticed some one reproduce your results in v0.2.0, but i do not find the git branch to run with OpenPCDet v0.2.0. could you please help me to reproduce the performance in your paper.

Hello, thanks a lot for your clear and great code. I have tried to reproduce the domain adaptation result from waymo to kitti. However, I found that it cost me nearly 3h/epoch with 4 3080. Should the self-training be as fast as training on naive OpenPCDet? Or the memory ensembling costs that much? Hope to get your reply.

I did the experiments on Waymo -> MyOwnDataset of ['Vehicle', 'Pedestrian', 'Cyclist'] on PVRCNN. But the result shows, after self-train, the AP of the model for 3 classes is worse than that of the model just train for single. How can I get a stable model for all 3 classes?

hi, Jihan I have try a lot of times using your lastest config files, but fail to reproduce the performance of ST from waymo to nus. for secondiou_st, the best performance showed in No. 4 epoch. [2021-12-02 09:53:08,270 eval_utils.py 50 INFO] *************** EPOCH 4 EVALUATION ***************** [2021-12-02 09:54:26,908 eval_utils.py 89 INFO] *************** Performance of EPOCH 4 ***************** [2021-12-02 09:54:26,908 eval_utils.py 91 INFO] Generate label finished(sec_per_example: 0.0131 second). [2021-12-02 09:54:26,908 eval_utils.py 107 INFO] recall_roi_0.3: 0.582705 [2021-12-02 09:54:26,908 eval_utils.py 108 INFO] recall_rcnn_0.3: 0.582705 [2021-12-02 09:54:26,909 eval_utils.py 107 INFO] recall_roi_0.5: 0.476606 [2021-12-02 09:54:26,909 eval_utils.py 108 INFO] recall_rcnn_0.5: 0.476606 [2021-12-02 09:54:26,909 eval_utils.py 107 INFO] recall_roi_0.7: 0.234905 [2021-12-02 09:54:26,909 eval_utils.py 108 INFO] recall_rcnn_0.7: 0.234905 [2021-12-02 09:54:26,911 eval_utils.py 116 INFO] Average predicted number of objects(6019 samples): 5.036 [2021-12-02 09:54:37,770 eval_utils.py 127 INFO] Car [email protected], 0.70, 0.70: bbox AP:45.3321, 45.3321, 45.3321 bev AP:30.9359, 30.9359, 30.9359 3d AP:17.2579, 17.2579, 17.2579 aos AP:26.60, 26.60, 26.60 Car [email protected], 0.70, 0.70: bbox AP:47.2941, 47.2941, 47.2941 bev AP:26.0944, 26.0944, 26.0944 3d AP:12.9529, 12.9529, 12.9529 aos AP:27.46, 27.46, 27.46 Car [email protected], 0.50, 0.50: bbox AP:45.3321, 45.3321, 45.3321 bev AP:34.6136, 34.6136, 34.6136 3d AP:33.8733, 33.8733, 33.8733 aos AP:26.60, 26.60, 26.60 Car [email protected], 0.50, 0.50: bbox AP:47.2941, 47.2941, 47.2941 bev AP:34.4677, 34.4677, 34.4677 3d AP:29.9190, 29.9190, 29.9190 aos AP:27.46, 27.46, 27.46 I used 4 RTX2080TI cards, with batch_size 16. Could you help me to find what is wrong with it?

I noticed that, in your ST3D++ paper, the transformation parameters γ and β are shared between two domains since "the transformation parameters γ and β are domain agnostic". but in paper "Domain-specific batch normalization for unsupervised domain adaptation", γ and β are not shared between source domain and target domain. I want to confirm whether the γ and β are shared to reproduce you work. Look forward to you reply, thank you very much!

hello，jihan. Thanks for your work, when I use the pretrained model to eval the KITTI results, I got a bad results as fellows: [2021-07-30 10:39:38,642 eval_utils.py 91 INFO] *************** Performance of EPOCH 80 ***************** [2021-07-30 10:39:38,643 eval_utils.py 93 INFO] Generate label finished(sec_per_example: 0.0255 second). [2021-07-30 10:39:38,643 eval_utils.py 109 INFO] recall_roi_0.3: 0.977685 [2021-07-30 10:39:38,643 eval_utils.py 110 INFO] recall_rcnn_0.3: 0.977685 [2021-07-30 10:39:38,643 eval_utils.py 109 INFO] recall_roi_0.5: 0.798957 [2021-07-30 10:39:38,643 eval_utils.py 110 INFO] recall_rcnn_0.5: 0.798957 [2021-07-30 10:39:38,643 eval_utils.py 109 INFO] recall_roi_0.7: 0.070421 [2021-07-30 10:39:38,643 eval_utils.py 110 INFO] recall_rcnn_0.7: 0.070421 [2021-07-30 10:39:38,647 eval_utils.py 118 INFO] Average predicted number of objects(3769 samples): 11.060 [2021-07-30 10:39:57,848 eval_utils.py 129 INFO] Car [email protected], 0.70, 0.70: bbox AP:17.6470, 23.7820, 33.4059 bev AP:11.3823, 14.4799, 21.0351 3d AP:0.6494, 0.5051, 9.0909 aos AP:17.53, 23.37, 32.83 Car [email protected], 0.70, 0.70: bbox AP:18.4800, 23.7817, 28.3788 bev AP:11.2770, 13.3130, 14.3258 3d AP:0.0528, 0.1354, 0.1768 aos AP:18.36, 23.37, 27.70 Car [email protected], 0.50, 0.50: bbox AP:17.6470, 23.7820, 33.4059 bev AP:45.1927, 54.1058, 63.2144 3d AP:32.6627, 41.2533, 48.3045 aos AP:17.53, 23.37, 32.83 Car [email protected], 0.50, 0.50: bbox AP:18.4800, 23.7817, 28.3788 bev AP:47.9240, 55.8623, 62.0041 3d AP:33.6433, 41.2521, 45.2606 aos AP:18.36, 23.37, 27.70

And I just use the pretrianed second_iou model with kitti_models/secondiou_orcale config.

Hi, Jihan, thanks for open source the paper, I just have some question about tables in paper. why do experiment in Table S11,S12,S13? Those experiments are evaluated use IOU=0.5,but before in Table 2,3,4 , you already do those experiment but with IOU=0.7? What's more, In Table S11, why it has opposite result with Table 2, in Table S11, source only is better that ROS, but in Table 2, ros is better than source only.

Thanks for reply.