I trained resnet-50 on Cityscapes with one gpu for three times that only different on the strategy of weight decay. The other parameters are the same as in your example. When I apply L2-SP regularization, the precision on val_set is 69.93 mIoU. When I apply L2 regularization, the precision on val_set is 69.68 mIoU. When I apply No regularization, the precision on val_set is 72.15 mIoU. So, my question is does loss regularization really work? Why the highest accuracy occurs with no loss regularization? Or how can I change the hyper-parameters to improve the accuracy of L2-SP regularization?

Hi, thanks for your nice work！ Recently, I am following this pspnet work. I am curious about how you trained on the fine + coarse data set. Here is my idea:

1. split training into 2 steps: train on the coarse or fine data set first and on the another then.
2. mix fine and coarse data set, and train once only.

Recently,I'm learning about semantic segmentation, and I'm glad to learn your Repositories: pspnet-tf-fiction on github.This is a particularly good reproduction. And I have a few questions for you:

1. How do you import the pre-training weight of resnet_v1_101 into the model? I think the code is initialized with 'he'.
2. What is the role of 'auxiliary loss operations' in the model? When I use resnet_v1_50.ckpt, it will report an error. When I screen it out, the output loss will be larger or smaller. I have not learned enough about this part, and there are many things to learn from you. Your help is very important to me, and I am looking forward to your reply.

model not loading. It gives an error. Below is the error.

Tensor name "resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/BatchNorm/beta" not found in checkpoint files /PSPNET/PSPNet-TF-Reproduce/z_pretrained_weights/resnet_v1_101.ckpt [[node save/RestoreV2 (defined at inference.py:71) = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

What could be the problem? I downloaded the model from .sh file given in the code.

< Finetuning Process: not import resnet_v1_101/psp/pool1/BatchNorm/gamma:0 > < Finetuning Process: not import resnet_v1_101/block4/unit_4/weights:0 > < Finetuning Process: not import resnet_v1_101/block4/unit_4/BatchNorm/beta:0 > < Finetuning Process: not import resnet_v1_101/block4/unit_4/BatchNorm/gamma:0 > < Finetuning Process: not import resnet_v1_101/logits/weights:0 > < Finetuning Process: not import resnet_v1_101/logits/biases:0 > < Succesfully loaded fine-tune model from /nfs/private/PSPNet-TF-Reproduce-master/z_pretrained_weights/resnet_v1_101.c kpt. >

< training process begins >

loss or weight norm is nan. Training Stopped!

Thank you for reproducing pspnet. I find you set num_classes = 150 instead of 151 while 0 represents 'other' class. I think this may cause a little difference during training and testing.

I tried the sync bn proposed by your codes and find that the memory cost increase tremendously. My experimental environment includes 4 Titan X GPU that could fit 5 batches per gpu when using Deeplab3+ as the solution, while I could only fit 1 batch per gpu after adopting sync bn. Also the running time is increased also.

Could this be caused by the absence of NVLink on my severs?

Hi, I try to learn to train pspnet on cityscapes(only train dataset), but after several iters , always ResourceExhaustedError, I try to reduce batch_size and train_size from 864 to 512.and I use 4*TITAN-XP(12G). Could you give to some advice? Thank you so much! python ./run.py --network 'resnet_v1_101' --visible_gpus '0,1,2,3' --reader_method 'queue' --batch_size 4 --poly_lr 1 --lrn_rate 0.01 --momentum 0.9 --weight_decay_mode 0 --weight_decay_rate 0.0001 --weight_decay_rate2 0.001 --database 'Cityscapes' --subsets_for_training 'train' --train_image_size 512 --snapshot 10000 --train_max_iter 50000 --test_image_size 512 --random_rotate 0 --fine_tune_filename './z_pretrained_weights/resnet_v1_101.ckpt'

I have converted my dataset as ADE format annotation images.

I have only two classes. like wise i will have only two pixel values 1 and 2 in annotation image rest all pixels will have 0.

I have used this command

python ./run.py --network 'resnet_v1_50' --visible_gpus '0,1' --reader_method 'queue' --lrn_rate 0.0001 --weight_decay_mode 0 --weight_decay_rate 0.0001 --weight_decay_rate2 0.001 --database 'ADE' --subsets_for_training 'train' --batch_size 2 --train_image_size 480 --snapshot 30000 --train_max_iter 90000 --test_image_size 480 --random_rotate 0 --fine_tune_filename './z_pretrained_weights/resnet_v1_50.ckpt'

After some iterations (650) , I am getting following error

loss or weight norm is nan. Training Stopped!

I have seen issue #15 , but not worked.

I think data set representation would be wrong. like number of classes.

is there any way to check my custom data set in correct ADE format or not.

Please help me out, Thanks

I tried to train on ADE dataset, but I still met the error proposed in #15 . There are two differences with the example script (3.b):

1. I used --batch_size 2 --gpu_num 4 because of GPU memory limitation. But I decrease the --lrn_rate to 0.00001 as suggested in #15 .

2. I used resnet_v1_101 network and resnet_v1_101.ckpt as the pretrained model.

My Tensorflow is 1.8.0. Any idea about this error? Thanks!

            if 'train' in stats_mode:
                xn = tf.nn.batch_normalization(
                    list_input[i], mean, var, beta, gamma, bn_epsilon)
                if tf.get_variable_scope().reuse or 'gather' not in stats_mode:
                    list_output.append(xn)
                else:
                    # gather stats and it is the main gpu device.
                    xn = update_bn_ema(xn, mean, var, moving_mean, moving_var, bn_ema)
                    list_output.append(xn)
            else:
                xn = tf.nn.batch_normalization(
                    list_input[i], moving_mean, moving_var, beta, gamma, bn_epsilon)
                list_output.append(xn)

Hello! I've found a performance issue in /reader.py: dataset.batch(batch_size)(here) should be called before .map(_training_data_preprocess, num_parallel_calls=batch_size)(here), which could make your program more efficient.

Here is the tensorflow document to support it.

Besides, you need to check the function _training_data_preprocess called in .map(_training_data_preprocess, num_parallel_calls=batch_size) whether to be affected or not to make the changed code work properly. For example, if _training_data_preprocess needs data with shape (x, y, z) as its input before fix, it would require data with shape (batch_size, x, y, z) after fix.

Looking forward to your reply. Btw, I am very glad to create a PR to fix it if you are too busy.

Hi, I am new to segmentation and trying to use this repo to train the network from scratch on a custom dataset having only 2 classes (binary segmentation) and so if you could please guide me as to where do i need to make the changes in the code. Also is it possible to train if a dataset has images in 4 channels or 1 channel and not 3. A help would be highly appreciated. Thanks!!