Training log. [2019-06-14 18:10:31,725 INFO train.py line 154 10613] arch: psp aux_weight: 0.4 base_lr: 0.01 base_size: 2048 batch_size: 16 batch_size_val: 1 classes: 19 colors_path: dataset/cityscapes/cityscapes_colors.txt data_root: datasets/cityscapes/ dist_backend: nccl dist_url: tcp://127.0.0.1:6789 distributed: True epochs: 200 evaluate: False has_prediction: False ignore_label: 255 index_split: 5 index_start: 0 index_step: 0 keep_batchnorm_fp32: None layers: 50 loss_scale: None manual_seed: None model_path: exp/cityscapes/pspnet50/model/train_epoch_200.pth momentum: 0.9 multiprocessing_distributed: True names_path: dataset/cityscapes/cityscapes_names.txt ngpus_per_node: 8 opt_level: O0 power: 0.9 print_freq: 10 rank: 0 resume: None rotate_max: 10 rotate_min: -10 save_folder: exp/cityscapes/pspnet50/result/epoch_200/val/ss save_freq: 1 save_path: exp/cityscapes/pspnet50/model scale_max: 2.0 scale_min: 0.5 scales: [1.0] split: val start_epoch: 0 sync_bn: True test_gpu: [0] test_h: 713 test_list: semseg-master/dataset/cityscapes/fine_val.txt test_w: 713 train_gpu: [0, 1, 2, 3, 4, 5, 6, 7] train_h: 713 train_list: semseg-master/dataset/cityscapes/fine_train.txt train_w: 713 use_apex: True val_list: semseg-master/dataset/cityscapes/fine_val.txt weight: None weight_decay: 0.0001 workers: 16 world_size: 8 zoom_factor: 8

Test log [2019-06-16 08:08:38,259 INFO test.py line 249 23114] Eval result: mIoU/mAcc/allAcc 0.7695/0.8400/0.9603. [2019-06-16 08:08:38,259 INFO test.py line 251 23114] Class_0 result: iou/accuracy 0.9804/0.9881, name: road. [2019-06-16 08:08:38,259 INFO test.py line 251 23114] Class_1 result: iou/accuracy 0.8454/0.9255, name: sidewalk. [2019-06-16 08:08:38,259 INFO test.py line 251 23114] Class_2 result: iou/accuracy 0.9235/0.9677, name: building. [2019-06-16 08:08:38,259 INFO test.py line 251 23114] Class_3 result: iou/accuracy 0.5557/0.6280, name: wall. [2019-06-16 08:08:38,259 INFO test.py line 251 23114] Class_4 result: iou/accuracy 0.6037/0.6966, name: fence. [2019-06-16 08:08:38,260 INFO test.py line 251 23114] Class_5 result: iou/accuracy 0.6419/0.7436, name: pole. [2019-06-16 08:08:38,260 INFO test.py line 251 23114] Class_6 result: iou/accuracy 0.7032/0.8118, name: traffic light. [2019-06-16 08:08:38,260 INFO test.py line 251 23114] Class_7 result: iou/accuracy 0.7856/0.8581, name: traffic sign. [2019-06-16 08:08:38,260 INFO test.py line 251 23114] Class_8 result: iou/accuracy 0.9260/0.9675, name: vegetation. [2019-06-16 08:08:38,260 INFO test.py line 251 23114] Class_9 result: iou/accuracy 0.6553/0.7457, name: terrain. [2019-06-16 08:08:38,260 INFO test.py line 251 23114] Class_10 result: iou/accuracy 0.9460/0.9770, name: sky. [2019-06-16 08:08:38,260 INFO test.py line 251 23114] Class_11 result: iou/accuracy 0.8237/0.9218, name: person. [2019-06-16 08:08:38,260 INFO test.py line 251 23114] Class_12 result: iou/accuracy 0.6327/0.7595, name: rider. [2019-06-16 08:08:38,260 INFO test.py line 251 23114] Class_13 result: iou/accuracy 0.9490/0.9782, name: car. [2019-06-16 08:08:38,260 INFO test.py line 251 23114] Class_14 result: iou/accuracy 0.7279/0.7737, name: truck. [2019-06-16 08:08:38,260 INFO test.py line 251 23114] Class_15 result: iou/accuracy 0.8610/0.9364, name: bus. [2019-06-16 08:08:38,260 INFO test.py line 251 23114] Class_16 result: iou/accuracy 0.6388/0.6618, name: train. [2019-06-16 08:08:38,260 INFO test.py line 251 23114] Class_17 result: iou/accuracy 0.6448/0.7306, name: motorcycle. [2019-06-16 08:08:38,260 INFO test.py line 251 23114] Class_18 result: iou/accuracy 0.7760/0.8881, name: bicycle.

By the way, training time is much longer than your report. ( i use 8 P40)

Totally 20210 samples in train set.
Starting Checking image&label pair train list...
Checking image&label pair train list done!
Traceback (most recent call last):
  File "tool/train.py", line 426, in <module>
    main()
  File "tool/train.py", line 107, in main
    main_worker(args.train_gpu, args.ngpus_per_node, args)
  File "tool/train.py", line 236, in main_worker
    loss_train, mIoU_train, mAcc_train, allAcc_train = train(train_loader, model, optimizer, epoch)
  File "tool/train.py", line 281, in train
    output, main_loss, aux_loss = model(input, target)
  File "/home/xrlin/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/xrlin/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/home/xrlin/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/xrlin/Desktop/semseg/model/pspnet.py", line 91, in forward
    x = self.layer4(x_tmp)
  File "/home/xrlin/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/xrlin/anaconda3/lib/python3.7/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/home/xrlin/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/xrlin/Desktop/semseg/model/resnet.py", line 87, in forward
    out = self.bn3(out)
  File "/home/xrlin/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/xrlin/anaconda3/lib/python3.7/site-packages/torch/nn/modules/batchnorm.py", line 81, in forward
    exponential_average_factor, self.eps)
  File "/home/xrlin/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 1656, in batch_norm
    training, momentum, eps, torch.backends.cudnn.enabled
RuntimeError: CUDA out of memory. Tried to allocate 450.00 MiB (GPU 0; 23.65 GiB total capacity; 22.42 GiB already allocated; 105.12 MiB free; 311.16 MiB cached)

I tried to run every configuration in this project with one GPU (Titan RTX with 24GB memory), but it shows CUDA out of memory, which is wired because there are no other programs using the GPU. Do you have any suggestions for this issue?

When trying to use multithread, the following error occurs:

OMP: Hint Please submit a bug report with this message, compile and run commands used, and machine configuration info including native compiler and ope│··································
rating system versions. Faster response will be obtained by including all program sources. For information on submitting this issue, please see http://│··································
www.intel.com/software/products/support/.                                                                                                              │··································
OMP: Error #13: Assertion failure at z_Linux_util.cpp(2361).                                                                                           │··································
OMP: Hint Please submit a bug report with this message, compile and run commands used, and machine configuration info including native compiler and ope│··································
rating system versions. Faster response will be obtained by including all program sources. For information on submitting this issue, please see http://│··································
www.intel.com/software/products/support/.                                                                                                              │··································
OMP: Error #13: Assertion failure at z_Linux_util.cpp(2361).                                                                                           │··································
OMP: Hint Please submit a bug report with this message, compile and run commands used, and machine configuration info including native compiler and ope│··································
rating system versions. Faster response will be obtained by including all program sources. For information on submitting this issue, please see http://│··································
www.intel.com/software/products/support/.                                                                                                              │··································
OMP: Error #13: Assertion failure at z_Linux_util.cpp(2361).                                                                                           │··································
OMP: Hint Please submit a bug report with this message, compile and run commands used, and machine configuration info including native compiler and ope│··································
rating system versions. Faster response will be obtained by including all program sources. For information on submitting this issue, please see http://│··································
www.intel.com/software/products/support/.                                                                                                              │··································
OMP: Error #13: Assertion failure at z_Linux_util.cpp(2361).                                                                                           │··································
OMP: Hint Please submit a bug report with this message, compile and run commands used, and machine configuration info including native compiler and ope│··································
rating system versions. Faster response will be obtained by including all program sources. For information on submitting this issue, please see http://│··································
www.intel.com/software/products/support/.                                                                                                              │··································
OMP: Error #13: Assertion failure at z_Linux_util.cpp(2361).                                                                                           │··································
OMP: Hint Please submit a bug report with this message, compile and run commands used, and machine configuration info including native compiler and ope│··································
rating system versions. Faster response will be obtained by including all program sources. For information on submitting this issue, please see http://│··································
www.intel.com/software/products/support/.                                                                                                              │··································
OMP: Error #13: Assertion failure at z_Linux_util.cpp(2361).                                                                                           │··································
OMP: Hint Please submit a bug report with this message, compile and run commands used, and machine configuration info including native compiler and ope│··································
rating system versions. Faster response will be obtained by including all program sources. For information on submitting this issue, please see http://│··································
www.intel.com/software/products/support/.                                                                                                              │··································
OMP: Error #13: Assertion failure at z_Linux_util.cpp(2361).                                                                                           │··································
OMP: Hint Please submit a bug report with this message, compile and run commands used, and machine configuration info including native compiler and ope│··································
rating system versions. Faster response will be obtained by including all program sources. For information on submitting this issue, please see http://│··································
www.intel.com/software/products/support/.                                                                                                              │··································
Traceback (most recent call last):                                                                                                                     │··································
  File "tool/train.py", line 456, in <module>                                                                                                          │··································
    main()                                                                                                                                             │··································
  File "tool/train.py", line 106, in main                                                                                                              │··································
    mp.spawn(main_worker, nprocs=args.ngpus_per_node, args=(args.ngpus_per_node, args))                                                                │··································
  File "/home/lzx/.conda/envs/pytorchseg10/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn                             │··································
    while not spawn_context.join():                                                                                                                    │··································
  File "/home/lzx/.conda/envs/pytorchseg10/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 107, in join                              │··································
    (error_index, name)                                                                                                                                │··································
Exception: process 2 terminated with signal SIGABRT

This will happen when multiprocessing_distributed and use_apex are True. Configuration details:

TRAIN:
  arch: psp
  layers: 101
  sync_bn: False  # adopt syncbn or not
  train_h: 585
  train_w: 585
  scale_min: 0.5  # minimum random scale
  scale_max: 2.0  # maximum random scale
  rotate_min: -10  # minimum random rotate
  rotate_max: 10  # maximum random rotate
  zoom_factor: 8  # zoom factor for final prediction during training, be in [1, 2, 4, 8]
  ignore_label: 255
  aux_weight: 0.4
  train_gpu: [0, 1, 2, 3, 4, 5, 6, 7]
  workers: 16  # data loader workers
  batch_size: 16  # batch size for training
  batch_size_val: 8  # batch size for validation during training, memory and speed tradeoff
  base_lr: 0.01
  epochs: 200
  start_epoch: 0
  new_epoch: 300  # for resume, how many new epochs to train
  power: 0.9
  momentum: 0.9
  weight_decay: 0.0001
  manual_seed: 520
  print_freq: 10
  save_freq: 1
  save_path: exp/cityscapes/pspnet101/model
  weight:  # path to initial weight (default: none)
  resume:  # path to latest checkpoint (default: none)
  evaluate: True # evaluate on validation set, extra gpu memory needed and small batch_size_val is recommend
Distributed:
  dist_url: tcp://127.0.0.1:6789
  dist_backend: 'nccl'
  multiprocessing_distributed: True
  world_size: 1
  rank: 0
  use_apex: True
  opt_level: 'O0'
  keep_batchnorm_fp32:
  loss_scale:

I have tried to do some search but they are not working.

Could you please give me some advice?

Thanks a lot!

Hi, I'm trying to use your fantastic code but encountered something confused when I try to load a checkpoint of 200 epoch training.

The mIoU of training reached 0.739 at epoch 200. But when I try to load this checkpoint and continually training, the performance dropped from 0.739 to 0.563 at epoch 201.

It's quite hard for me to understand this phenomenon, could you give me some advices? Thank you so much!

Here is my config file:

DATA:
  data_root: /datasets-ssd/cityscapes
  train_list: dataset/cityscapes/list/fine_train.txt
  val_list: dataset/cityscapes/list/fine_val.txt
  classes: 19

TRAIN:
  arch: psp
  layers: 101
  sync_bn: True  # adopt syncbn or not
  train_h: 513
  train_w: 513
  scale_min: 0.5  # minimum random scale
  scale_max: 2.0  # maximum random scale
  rotate_min: -10  # minimum random rotate
  rotate_max: 10  # maximum random rotate
  zoom_factor: 8  # zoom factor for final prediction during training, be in [1, 2, 4, 8]
  ignore_label: 255
  aux_weight: 0.4
#  train_gpu: [0, 1, 2, 3, 4, 5, 6, 7]
  train_gpu: [0]
  workers: 6  # data loader workers
  batch_size: 2  # batch size for training
  batch_size_val: 1  # batch size for validation during training, memory and speed tradeoff
  base_lr: 0.01
  epochs: 400
  start_epoch: 200
  power: 0.9
  momentum: 0.9
  weight_decay: 0.0001
  manual_seed: 520
  print_freq: 10
  save_freq: 1
  save_path: exp/cityscapes/pspnet101/model
  weight:  # path to initial weight (default: none)
  resume:  exp/cityscapes/pspnet101/model/train_epoch_200.pth # path to latest checkpoint (default: none)
  evaluate: False  # evaluate on validation set, extra gpu memory needed and small batch_size_val is recommend
Distributed:
  distributed: True
  dist_url: tcp://127.0.0.1:6789
  dist_backend: 'nccl'
  multiprocessing_distributed: True
  world_size: 1
  rank: 0
  use_apex: True
  opt_level: 'O0'
  keep_batchnorm_fp32:
  loss_scale:

TEST:
  test_list: dataset/cityscapes/list/fine_val.txt
  split: val  # split in [train, val and test]
  base_size: 2048  # based size for scaling
  test_h: 713
  test_w: 713
  scales: [1.0]  # evaluation scales, ms as [0.5, 0.75, 1.0, 1.25, 1.5, 1.75]
  has_prediction: False  # has prediction already or not
  index_start: 0  # evaluation start index in list
  index_step: 0  # evaluation step index in list, 0 means to end
  test_gpu: [0]
  model_path: exp/cityscapes/pspnet101/model/train_epoch_400.pth  # evaluation model path
  save_folder: exp/cityscapes/pspnet101/result/epoch_400/val/ss  # results save folder
  colors_path: dataset/cityscapes/cityscapes_colors.txt  # path of dataset colors
  names_path: dataset/cityscapes/cityscapes_names.txt  # path of dataset category names

My environment:

• 2080ti, 12G
• CUDA 10.0
• python 3.6

Hoping for your reply! Thank you again!

I cant' understand the mean about follow code:

    def scale_process(self, image, crop_h, crop_w, h, w, mean, std=None, stride_rate=2 / 3):
        ori_h, ori_w, _ = image.shape
        # 填充周边
        pad_h = max(crop_h - ori_h, 0)
        pad_w = max(crop_w - ori_w, 0)
        pad_h_half = int(pad_h / 2)
        pad_w_half = int(pad_w / 2)
        if pad_h > 0 or pad_w > 0:
            image = cv2.copyMakeBorder(image, pad_h_half, pad_h - pad_h_half, pad_w_half, pad_w - pad_w_half,
                                       cv2.BORDER_CONSTANT, value=mean)
        new_h, new_w, _ = image.shape
        # FAQ:
        stride_h = int(np.ceil(crop_h * stride_rate))
        stride_w = int(np.ceil(crop_w * stride_rate))
        grid_h = int(np.ceil(float(new_h - crop_h) / stride_h) + 1)
        grid_w = int(np.ceil(float(new_w - crop_w) / stride_w) + 1)
        prediction_crop = np.zeros((new_h, new_w, self.numclass), dtype=float)
        count_crop = np.zeros((new_h, new_w), float)
        for index_h in range(0, grid_h):
            for index_w in range(0, grid_w):
                s_h = index_h * stride_h
                e_h = min(s_h + crop_h, new_h)
                s_h = e_h - crop_h
                s_w = index_w * stride_w
                e_w = min(s_w + crop_w, new_w)
                s_w = e_w - crop_w
                image_crop = image[s_h:e_h, s_w:e_w].copy()
                count_crop[s_h:e_h, s_w:e_w] += 1
                prediction_crop[s_h:e_h, s_w:e_w, :] += self.net_process(image_crop, mean, std)
        prediction_crop /= np.expand_dims(count_crop, 2)
        ..........

i know that it need to process with the train height、width,But I dont't understand! Appreciate with any suggest!

I have tried to train PSPNet50 and PSANet50. PSPNet has been successfully trained, but PSANet is stuck in building up model. Has anyone encountered similar problem?

The GPU I used is RTX 8000 with 48G. However, I cannot run the it successfully and the error is "cuDNN error: CUDNN_STATUS_INTERNAL_ERROR". I notice that this appears because of the pspnet needs a more powerful GPU. I tried to decrease the batch size and resolution, and I find out that the occupation of GPU rise to a relative large valve and then decrease to about 11G. I speculate that the reason I fail to run the pspnet is the rapid rise of the occupation of GPU. The occupation is rather large (about 40G) even I set the batch size as 1. How can I address this?

Hi,

we are training Pascal VOC 2012 trainaug with PSPnet50. We have only 4 gpus, but the training time is 10x more, which is strange (we could even fit more into a batch).

our setup:

• 4 x P100 with 16GB
• batch size:20
• lr: 0.005

Is DDP enabled by default? What else could cause the slowing down? Only having 2x less GPUs is not proportionate with the speed drop.

@zhixuanli you got close with cityscapes in https://github.com/hszhao/semseg/issues/7#issuecomment-536251008 but still not the same. Did you only change the GPUs? Is that training time with distributed data parallel model?

@hszhao

thx you code.when i input ‘sh tool/train.sh ade20k pspnet50’，i want to train pspnet50，but display the results of test .i don't know how change？Can you help me? /semseg/config/ade20k/ade20k_pspnet101.yaml DATA: data_root: /DATA/maran/anaconda3/Project/semseg/data/ADEChallengeData2016/images/data/ train_list: /DATA/maran/anaconda3/Project/semseg/data/ADEChallengeData2016/training.txt val_list: /DATA/maran/anaconda3/Project/semseg/data/ADEChallengeData2016/validation.txt classes: 150

TRAIN: arch: psp layers: 50 sync_bn: True # adopt sync_bn or not train_h: 473 train_w: 473 scale_min: 0.5 # minimum random scale scale_max: 2.0 # maximum random scale rotate_min: -10 # minimum random rotate rotate_max: 10 # maximum random rotate zoom_factor: 8 # zoom factor for final prediction during training, be in [1, 2, 4, 8] ignore_label: 255 aux_weight: 0.4 train_gpu: [0, 1, 2, 3, 4, 5, 6, 7] workers: 16 # data loader workers batch_size: 16 # batch size for training batch_size_val: 8 # batch size for validation during training, memory and speed tradeoff base_lr: 0.01 epochs: 200 start_epoch: 101 power: 0.9 momentum: 0.9 weight_decay: 0.0001 manual_seed: print_freq: 10 save_freq: 1 save_path: exp/ade20k/pspnet50/model weight: # path to initial weight (default: none) resume: exp/ade20k/pspnet50/model/train_epoch_100.pth # path to latest checkpoint (default: none) evaluate: False # evaluate on validation set, extra gpu memory needed and small batch_size_val is recommend Distributed: dist_url: tcp://127.0.0.1:6789 dist_backend: 'nccl' multiprocessing_distributed: True world_size: 1 rank: 0 use_apex: True opt_level: 'O0' keep_batchnorm_fp32: loss_scale:

TEST: test_list: /DATA/maran/anaconda3/Project/semseg/data/ADEChallengeData2016/validation.txt split: val # split in [train, val and test] base_size: 512 # based size for scaling test_h: 473 test_w: 473 scales: [1.0] # evaluation scales, ms as [0.5, 0.75, 1.0, 1.25, 1.5, 1.75] has_prediction: False # has prediction already or not index_start: 0 # evaluation start index in list index_step: 0 # evaluation step index in list, 0 means to end test_gpu: [0] model_path: exp/ade20k/pspnet50/model/train_epoch_100.pth # evaluation model path save_folder: exp/ade20k/pspnet50/result/epoch_100/val/ss # results save folder colors_path: dataset/ade20k/ade20k_colors.txt # path of dataset colors names_path: dataset/ade20k/ade20k_names.txt # path of dataset category names

Hi, hs. In your code,

if args.distributed: torch.cuda.set_device(gpu) args.batch_size = int(args.batch_size / ngpus_per_node) args.batch_size_val = int(args.batch_size_val / ngpus_per_node) args.workers = int(args.workers / ngpus_per_node)

I think the default batch_size_val should be as the same as the ngpus_per_node, or get an error: ValueError: batch_size should be a positive integeral value, but got batch_size=0

Thank you for your codes. I did not find cityscapes_colors.txt and cityscapes_names.txt described in .yaml file in the google drive. What are their format?

THX and have a nice day!

https://github.com/hszhao/semseg/blob/4f274c3f276778228bc14a4565822d46359f0cc8/util/util.py#L63 This line of code works if there are non-zero instances of intersection between the output and target. There should be a check for this.

File "/content/gdrive/My Drive/Project/semseg/util/dataset.py", line 64, in getitem image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) # convert cv2 read image from BGR order to RGB order cv2.error: OpenCV(4.5.4-dev) /tmp/pip-req-build-v2fnlv3a/opencv/modules/imgproc/src/color.cpp:182: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'

aparently the code can't open the ADE dataset, i've placed on ade20k_pspnet50.yaml data_root the path to ADE Dataset, inside the folder with the following items "ADE (containing training models and validation), index_ade20k.mat, index_ade20k.pkl, objects.txt" what could I be doing wrong?

Dr. Zhao, in your PSA paper, the model adopts ResNet-FCN-backbone. But I find no code about confusion about feature maps of different scales. Is my understanding right? Thank you very much.

I am using test.py and it perfectly renders this color coded image. After playing with the code I couldn't figure out how to output the individual segments as separate images. Any advice would be appreciated