Hello! This is a fairly stupid issue but it brings a lot of trouble. Can you please help me specify the path to checkpoint file? I tried everything including absolute path and specifying name of ckpt... I first copied checkpoint files to a folder 'model' and that should work with default settings in code, however it does not... Suggestions?

@hellochick

It is very impressive work. Thanks for sharing it I am using TiTanX pascal 12GB and I cannot train in your current setting. So, I wonder how much GPU did you use when training? I need reduce the batch size to 2 but the result is so bad

Second things, I do not want to use pre-train weight from caffe. Could we use pre-train from Imagenet with resnet from official website at https://github.com/zhengyang-wang/Deeplab-v2--ResNet-101--Tensorflow? I want to know how can we obtain the result without information from PSPNet caffe.

Thanks

Hi, I have a small question. I didn't find labeltrainId.png data that list in your "train_list.txt". The Cityscapes dataset only includes labelId.png. So are they same data? Since I used labelId.png instead, training model can not converge. Thank you.

It's a nice work, thanks for the code provided by the owner. And I notice the following the comment:

According from the prototxt in Caffe implement, learning rate must multiply by 10.0 in pyramid module.

I have found the Caffe implement by Hao Zhang, however I can't find the prototxt for training. How can I get it?

Thanks.

I resized images of cityscape dataset to 256x512 and I tried inference.py on them with the following command

python inference.py --img-path=./input/test.png

but it kept raising:

tensorflow.python.framework.errors_impl.InvalidArgumentError: input image must be of non-zero size
         [[Node: conv5_3_pool1_interp = ResizeBilinear[T=DT_FLOAT, align_corners=true, _device="/job:localhost/replica:0/task:0/cpu:0"](conv5_3_pool1_conv_bn/Relu, conv5_3_pool1_interp/size)]]

Can you please tell me how to run the inference code with images of different size?

i have selected some training images from ade20k dataset with their color codded _seg.png mask images i set certain parameters at first as the train.py

INPUT_SIZE = '473,473'                                  #"Comma-separated string with height and width of images."
LEARNING_RATE = 1e-3                                    #"Base learning rate for training with polynomial decay."
MOMENTUM = 0.9                                          #"Momentum component of the optimiser."
NUM_CLASSES = 150                                       #"Number of classes to predict (including background)."
NUM_STEPS = 60001                                       #"Number of training steps."
POWER = 0.9                                             #"Decay parameter to compute the learning rate."
RANDOM_SEED = 1234                                      #"Random seed to have reproducible results."
WEIGHT_DECAY = 0.0001                                   #"Regularisation parameter for L2-loss."
RESTORE_FROM = './model/ade20k/x/'                       #"Where restore model parameters from."
SNAPSHOT_DIR = './model/ade20k/x/'                     #"Where to save snapshots of the model."
SAVE_NUM_IMAGES = 4                                     #"How many images to save."
SAVE_PRED_EVERY = 50                                    #"Save summaries and checkpoint every often."
tf.reset_default_graph() 

but when i tried training code on them they give me the following error.

INFO:tensorflow:Restoring parameters from ./model/ade20k/x/model.ckpt-0
Restored model parameters from ./model/ade20k/x/model.ckpt-0
INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.ResourceExhaustedError'>, OOM when allocating tensor with shape[1333,2000,3] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[Node: create_inputs/Cast = Cast[DstT=DT_FLOAT, SrcT=DT_UINT8, _device="/job:localhost/replica:0/task:0/device:GPU:0"](create_inputs/concat)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Traceback (most recent call last):

  File "<ipython-input-7-78358ffb08d0>", line 1, in <module>
    runfile('D:/mobileNetPSPNet/tuning.py', wdir='D:/mobileNetPSPNet')

  File "C:\Users\BioHelwan\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 866, in runfile
    execfile(filename, namespace)

  File "C:\Users\BioHelwan\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 102, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "D:/mobileNetPSPNet/tuning.py", line 176, in <module>
    main()

  File "D:/mobileNetPSPNet/tuning.py", line 165, in main
    loss_value, _ = sess.run([reduced_loss, train_op], feed_dict=feed_dict)

  File "C:\Users\BioHelwan\Anaconda3\envs\tensorflow\Lib\site-packages\tensorflow\python\client\session.py", line 900, in run
    run_metadata_ptr)

  File "C:\Users\BioHelwan\Anaconda3\envs\tensorflow\Lib\site-packages\tensorflow\python\client\session.py", line 1135, in _run
    feed_dict_tensor, options, run_metadata)

  File "C:\Users\BioHelwan\Anaconda3\envs\tensorflow\Lib\site-packages\tensorflow\python\client\session.py", line 1316, in _do_run
    run_metadata)

  File "C:\Users\BioHelwan\Anaconda3\envs\tensorflow\Lib\site-packages\tensorflow\python\client\session.py", line 1335, in _do_call
    raise type(e)(node_def, op, message)

ResourceExhaustedError: OOM when allocating tensor with shape[3,3,256,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
	 [[Node: gradients/L2Loss_31_grad/mul = Mul[T=DT_FLOAT, _class=["loc:@gradients/AddN_37"], _device="/job:localhost/replica:0/task:0/device:GPU:0"](conv4_2_3x3/weights/read, mul/x)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

	 [[Node: conv4_3_3x3/weights/read/_741 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_321_conv4_3_3x3/weights/read", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.


Caused by op 'gradients/L2Loss_31_grad/mul', defined at:
  File "C:\Users\BioHelwan\Anaconda3\lib\site-packages\spyder\utils\ipython\start_kernel.py", line 223, in <module>
    main()
  File "C:\Users\BioHelwan\Anaconda3\lib\site-packages\spyder\utils\ipython\start_kernel.py", line 219, in main
    kernel.start()
  File "C:\Users\BioHelwan\Anaconda3\lib\site-packages\ipykernel\kernelapp.py", line 474, in start
    ioloop.IOLoop.instance().start()
  File "C:\Users\BioHelwan\Anaconda3\lib\site-packages\zmq\eventloop\ioloop.py", line 162, in start
    super(ZMQIOLoop, self).start()
  File "C:\Users\BioHelwan\Anaconda3\lib\site-packages\tornado\ioloop.py", line 887, in start
    handler_func(fd_obj, events)
  File "C:\Users\BioHelwan\Anaconda3\lib\site-packages\tornado\stack_context.py", line 275, in null_wrapper
    return fn(*args, **kwargs)
  File "C:\Users\BioHelwan\Anaconda3\lib\site-packages\zmq\eventloop\zmqstream.py", line 440, in _handle_events
    self._handle_recv()
  File "C:\Users\BioHelwan\Anaconda3\lib\site-packages\zmq\eventloop\zmqstream.py", line 472, in _handle_recv
    self._run_callback(callback, msg)
  File "C:\Users\BioHelwan\Anaconda3\lib\site-packages\zmq\eventloop\zmqstream.py", line 414, in _run_callback
    callback(*args, **kwargs)
  File "C:\Users\BioHelwan\Anaconda3\lib\site-packages\tornado\stack_context.py", line 275, in null_wrapper
    return fn(*args, **kwargs)
  File "C:\Users\BioHelwan\Anaconda3\lib\site-packages\ipykernel\kernelbase.py", line 276, in dispatcher
    return self.dispatch_shell(stream, msg)
  File "C:\Users\BioHelwan\Anaconda3\lib\site-packages\ipykernel\kernelbase.py", line 228, in dispatch_shell
    handler(stream, idents, msg)
  File "C:\Users\BioHelwan\Anaconda3\lib\site-packages\ipykernel\kernelbase.py", line 390, in execute_request
    user_expressions, allow_stdin)
  File "C:\Users\BioHelwan\Anaconda3\lib\site-packages\ipykernel\ipkernel.py", line 196, in do_execute
    res = shell.run_cell(code, store_history=store_history, silent=silent)
  File "C:\Users\BioHelwan\Anaconda3\lib\site-packages\ipykernel\zmqshell.py", line 501, in run_cell
    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
  File "C:\Users\BioHelwan\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 2717, in run_cell
    interactivity=interactivity, compiler=compiler, result=result)
  File "C:\Users\BioHelwan\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 2827, in run_ast_nodes
    if self.run_code(code, result):
  File "C:\Users\BioHelwan\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 2881, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-7-78358ffb08d0>", line 1, in <module>
    runfile('D:/mobileNetPSPNet/tuning.py', wdir='D:/mobileNetPSPNet')
  File "C:\Users\BioHelwan\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 866, in runfile
    execfile(filename, namespace)
  File "C:\Users\BioHelwan\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 102, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)
  File "D:/mobileNetPSPNet/tuning.py", line 176, in <module>
    main()
  File "D:/mobileNetPSPNet/tuning.py", line 125, in main
    grads = tf.gradients(reduced_loss, conv_trainable + fc_w_trainable + fc_b_trainable)
  File "C:\Users\BioHelwan\Anaconda3\envs\tensorflow\Lib\site-packages\tensorflow\python\ops\gradients_impl.py", line 532, in gradients
    gate_gradients, aggregation_method, stop_gradients)
  File "C:\Users\BioHelwan\Anaconda3\envs\tensorflow\Lib\site-packages\tensorflow\python\ops\gradients_impl.py", line 701, in _GradientsHelper
    lambda: grad_fn(op, *out_grads))
  File "C:\Users\BioHelwan\Anaconda3\envs\tensorflow\Lib\site-packages\tensorflow\python\ops\gradients_impl.py", line 396, in _MaybeCompile
    return grad_fn()  # Exit early
  File "C:\Users\BioHelwan\Anaconda3\envs\tensorflow\Lib\site-packages\tensorflow\python\ops\gradients_impl.py", line 701, in <lambda>
    lambda: grad_fn(op, *out_grads))
  File "C:\Users\BioHelwan\Anaconda3\envs\tensorflow\Lib\site-packages\tensorflow\python\ops\nn_grad.py", line 963, in _L2LossGrad
    return op.inputs[0] * grad
  File "C:\Users\BioHelwan\Anaconda3\envs\tensorflow\Lib\site-packages\tensorflow\python\ops\math_ops.py", line 847, in binary_op_wrapper
    return func(x, y, name=name)
  File "C:\Users\BioHelwan\Anaconda3\envs\tensorflow\Lib\site-packages\tensorflow\python\ops\math_ops.py", line 1091, in _mul_dispatch
    return gen_math_ops.mul(x, y, name=name)
  File "C:\Users\BioHelwan\Anaconda3\envs\tensorflow\Lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 5066, in mul
    "Mul", x=x, y=y, name=name)
  File "C:\Users\BioHelwan\Anaconda3\envs\tensorflow\Lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "C:\Users\BioHelwan\Anaconda3\envs\tensorflow\Lib\site-packages\tensorflow\python\framework\ops.py", line 3414, in create_op
    op_def=op_def)
  File "C:\Users\BioHelwan\Anaconda3\envs\tensorflow\Lib\site-packages\tensorflow\python\framework\ops.py", line 1740, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

...which was originally created as op 'L2Loss_31', defined at:
  File "C:\Users\BioHelwan\Anaconda3\lib\site-packages\spyder\utils\ipython\start_kernel.py", line 223, in <module>
    main()
[elided 20 identical lines from previous traceback]
  File "D:/mobileNetPSPNet/tuning.py", line 176, in <module>
    main()
  File "D:/mobileNetPSPNet/tuning.py", line 106, in main
    l2_losses = [WEIGHT_DECAY * tf.nn.l2_loss(v) for v in tf.trainable_variables() if 'weights' in v.name]
  File "D:/mobileNetPSPNet/tuning.py", line 106, in <listcomp>
    l2_losses = [WEIGHT_DECAY * tf.nn.l2_loss(v) for v in tf.trainable_variables() if 'weights' in v.name]
  File "C:\Users\BioHelwan\Anaconda3\envs\tensorflow\Lib\site-packages\tensorflow\python\ops\gen_nn_ops.py", line 4715, in l2_loss
    "L2Loss", t=t, name=name)
  File "C:\Users\BioHelwan\Anaconda3\envs\tensorflow\Lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "C:\Users\BioHelwan\Anaconda3\envs\tensorflow\Lib\site-packages\tensorflow\python\framework\ops.py", line 3414, in create_op
    op_def=op_def)
  File "C:\Users\BioHelwan\Anaconda3\envs\tensorflow\Lib\site-packages\tensorflow\python\framework\ops.py", line 1740, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[3,3,256,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
	 [[Node: gradients/L2Loss_31_grad/mul = Mul[T=DT_FLOAT, _class=["loc:@gradients/AddN_37"], _device="/job:localhost/replica:0/task:0/device:GPU:0"](conv4_2_3x3/weights/read, mul/x)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

	 [[Node: conv4_3_3x3/weights/read/_741 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_321_conv4_3_3x3/weights/read", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

how can i get rid of this error? and how can the train be done on the colored masks of ade20k dataset not gray masks? Thanks

Hi I'm running inference based on the model trained on ADE20K dataset and I would like to: (1) Derive the segmentation mask. i.e. matrix with same rows x cols as image but each cell reflected the class id prediction for the pixel (2) Get the corresponding ADE20K class name given a class id

Hi, @hellochick ,

I have performed the inference step after downloading the weight ckpt files you provided : python inference.py --img-path=./input/indoor_2.jpg --dataset cityscapes.

However, the result I got is very different to yours. (result link)

Would you suggest me how to solve this issue?

THANKS!

@hellochick Excuse me. Because only the TF 1.1 can run on our platform, I am wondering if this version can be used? Thank you for your help! Again, appreciation for your contribution Geng Qichuan

@hellochick Thank you for making your scripts public for the reference. I was wondering if these tensorflow version of PSPNet accommodates the training data with different size? My training data is [360X480X3] image and testing data is [360X480] grayscale image with class labelled as [0..N]. Again, appreciation for your contribution.

Bhadresh

i set index 0 to be ignored for ade20k training but in testing my trained model, i got class label 0 and person's label becomes 13 not 12 as in the check point model what can i do to fix this problem? and what is the expected labeling output for a model trained on the ade20k_2016challenge?

I have trained my PSP net model using this repo: https://github.com/holyseven/PSPNet-TF-Reproduce . I wanted to load it through your inference code.

I trained my for 3 classes on ADE_20K dataset.

I change inference.py code according to 3 classes.

I am getting following error:

NotFoundError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Key conv1_1_3x3_s2/weights not found in checkpoint
	 [[node save/RestoreV2 (defined at inference.py:105)  = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

Using yours inference.py , can't I load another model ? Is it model dependent ? It was trained on repo as suggested in readme.

Thanks

Hi,

I'm trying to train on cityscapes in order to first replicate the 70% miou and then move to other driving datasets to see what happens. However, I'm having trouble replicating this. I can't seem to get the loss to converge below 1.7. I'm training from scratch on purpose in order to get a clean baseline for the other datasets.

I have added seg_map= sess.run(raw_output_up) in inference.py line112 and get mask images. However it's not related to cityscapes original label ids. Could you tell me how can i get inference mask image, in which RGB values are correlating to cityscapes original label ids? @hellochick thank you a lot

If anyone has found a way to do this, could you please give me a hint? I guess there's simpler way than writing script to convert RGB segmentation output into these IDs (that could get complicated, since some classes have the same RGB -e.g. pole and pole group).

I tried to solve the problem with this code by getting IDs as: IDs = sess.run(raw_output_up) and then saving them, however this outputs an image with numbers which don't represent neither train IDs, nor class labels (numbers up to 255, but don't fit to these two IDs).... @jiaxinxie97 @Jason-xin @MichaelX99 @hellochick Thank you!