Hello, thank you for your excellent work. I am also interested in implementing PSPNet in TF. I have not tested your code yet, but about your output, I found some post process code in original matlab code.

function score = ms_caffe_process(input_data,net)
    score = net.forward({input_data});
    score = score{1};
    score_flip = net.forward({input_data(end:-1:1,:,:)});
    score_flip = score_flip{1};
    score = score + score_flip(end:-1:1,:,:);

    score = permute(score, [2 1 3]);
    score = exp(score);
    score = bsxfun(@rdivide, score, sum(score, 3));
end

It seems that they feed in the image twice each time, and one is flipped. Do you know why they did this?

Thank you

Hello,

Can anyone upload the .npy weights for cityscapes (and others if possible) since the conversion is not working for me, PSPNet caffe is not working well.

Thanks!

Hi Vlad,

I am trying to convert the original pspnet101_cityscapes weights from https://github.com/hszhao/PSPNet to keras.

python weight_converter.py pspnet101_cityscapes_713.prototxt pspnet101_cityscapes.caffemodel

In PSPNet-Keras-tensorflow I adjusted:

WEIGHTS = 'pspnet101_cityscapes.npy'
DATA_MEAN = np.array([[[72.78044, 83.21195, 73.45286]]]) # RGB
x = Conv2D(19, (1, 1), strides=(1, 1), name="conv6")(x) # changed the classes from 150 to 19 

PSPNet runs through without errors, unfortunately, the output for a test image remains indiscernible:

Do you have an idea what I could be missing?

Hello and thank you for your great work! I tried to do segmentation with this network (with cityscapes weights) and colours seem to be assigned completely wrong. Do you have an idea why this might be happening? I.e. vegetation is always blue, pavements red instead of magenta and people magenta instead of red, streets are black(static or unlabeled?) etc... Can it be that, for some reason, pixels are colourized by lowest scores? I was thinking about converting the weights myself, but I don't have pycaffe etc. installed. Could someone please upload the correct weights, after the issue with wrong order of parameters in conversion? @scenarios @karolmajek

I will post images later this evening, now I have to type from my phone and can't add pictures.

I would appreciate any suggestion what might be causing the issue. @wcy940418 @Vladkryvoruchko @jmtatsch

Hier ist what I'm using:

Name Version Build Channel

python 3.5.5 he025d50_2 conda-forge

cudatoolkit 8.0 4

cudnn 6.0 0

tensorflow 1.1.0 np112py35_0

tensorflow-gpu 1.1.0 np112py35_0

keras 2.0.6 py35_0 conda-forge

Hi @Vladkryvoruchko , I noticed you convert the parameter of bn layer in caffe to the numpy array by the order of 'mean = v[0], variance = v[1], scale = v[2], offset = v[3]' . However, the v[0-3] actually hold the value for scale, offset, mean, variance respectively according to the caffe blob's definition in bn layer.

The reason that you can still reproduce the results is that you reverse the order again when you try to restore the parameter for the keras model from the numpy array, i.e. model.get_layer(layer.name).set_weights([mean, variance, scale, offset]).

This may not hurt when using your code completely, but it is very unexpected for those people who just use the parameter converted by the code.

Thanks for your efforts!

Hi, when I run pspnet.py, I get a ValueError due to a negative dimension (output file with error pspnet.txt attached). I'm using virtualenv with python2.7, tensorflow version 1.3.0 and keras version 2.0.8. I'm also using the weight files that you've provided.

Any help would be greatly appreciated!

Thanks Nariman

In the readme, it says you can use the -ms and -s flags but these don't seem to be implemented and if you use them, you get an error saying as much.

Is it possible to use these options through the CLI?

I've created one virtual environment on Anaconda and installed tensorflow= 1.10.0 / numpy = 1.13.3 / scipy = 1.2.1 and keras.

I could not install tensorflow-gpu because I don't have gpu.. and If I run code, I'm getting this message saying, "from scipy.misc import imresize, imread ImportError: cannot import name 'imresize"

from somewhere, somebody said that 'imresize' is deprecated from scipy ==1.0.0 so I tried a lower version but the result was same too.

Do somebody have any idea to help me to resolve this problem?

I improved performance using cv2 for reading/resizing. I also used numpy instead of nested for in class_image_to_image Here is my fork: https://github.com/karolmajek/PSPNet-Keras-tensorflow

Now I am preparing YT vid:

Do you want me to make a pull request?

Hi, I am trying to get this running but I'm getting a Reason: Image not found error.

1. I created a virtualenv running python 3.6.6
2. I ran pip install -r requirements.txt --upgrade
3. I downloaded the weights and put them in the weights/keras dir
4. I ran python pspnet.py -m pspnet101_voc2012 -i example_images/pascal_voc.jpg -o example_results/pascal_voc.jpg

The image pascal_voc.jpg is in that that directory so I'm not sure why it can't find that image.

The console output I get is Using TensorFlow backend. and then it seems to fail when it imports keras

  File "pspnet.py", line 8, in <module>
    from keras import backend as K

It also says

Reason: image not found


Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/install_sources#common_installation_problems

Do i need to do other installation steps other than pip installing the dependencies?

In pspnet.py line 163, you are taking input size from args.

Next, in line 207, you are using that user defined input size to resize the image. cimg = misc.imresize(img, (args.input_size, args.input_size))

But now, you are not using the 'cimg' variable anywhere.

Furthermore, in line number 56, you are resizing the image again to an input shape which you have hardcoded for each pretrained model. (473x473 for pspnet101_voc2012) img = misc.imresize(img, self.input_shape)

So what's even the point of asking for size?

[Created Issue for tracking] Disclaimer: I know there are several repos on the Github, that already include PSPNet implemented in TensorFlow with all the features included, but aint give a shit about that, just want to help anyone who use this repo, and know that I have a high-quality code.

Alright, I haven't maintained anything here for a while, so its time to put some effort to solve as many issues as I can and upgrade repo to the:

• fix high-severity bugs
• latest Python/dependencies version
• Remove deprecated methods
• Optimize code
• Add some tests
• Maybe add some better deploy experience
• Add API layer(?)

I plan to deliver updates in several phases:

• [x] Write some simple smoke tests, to be sure system functioning Merged PRs: https://github.com/Vladkryvoruchko/PSPNet-Keras-tensorflow/pull/74

• [x] Resolve any issues that are related to the bugfix/performance/optimization using Python3.5 interpreter Merged PRs: https://github.com/Vladkryvoruchko/PSPNet-Keras-tensorflow/pull/58 https://github.com/Vladkryvoruchko/PSPNet-Keras-tensorflow/pull/37 https://github.com/Vladkryvoruchko/PSPNet-Keras-tensorflow/pull/70 https://github.com/Vladkryvoruchko/PSPNet-Keras-tensorflow/pull/71 https://github.com/Vladkryvoruchko/PSPNet-Keras-tensorflow/pull/72 https://github.com/Vladkryvoruchko/PSPNet-Keras-tensorflow/pull/73

• [ ] Remove blockers to be able to upgrade to the latest stable Python/TF/Keras versions Add support for Python3.4+, instead of some fixed version of interpreter

• [ ] I expect to have a bunch of new issues after upgrading to the new versions, so then will be the phase of solving new issues.

• [ ] Look and analyze https://github.com/Vladkryvoruchko/PSPNet-Keras-tensorflow/pull/41 (Make PSPNet fully convolutional)

• [ ] Refactor code OOP, structure(library API layer, CLI layer,).

• [ ] Create better logging

• [ ] Add evaluation script

• [ ] Upgrade training script

• [ ] Answer the most common questions in the READme, so fewer issues with common questions will be created

• [ ] (Different repository) Create Flask API wrapper around this project to ease up deploy/integration/whatever, so ppl without related experience can run it as easy as count to 3 e.g: git pull, make build; make deploy; then do simple http(s) request (docker-compose/Flask/nginx/uwsgi(or gunicorn)/celery worker/redis[or whatever configurable queue])

This PR will bring versions support to Python3.4+ and modern versions of TensorFlow/Keras

Ideally, PSPNet will run on every Python, starting from 3.4+

Issues(the same idea): https://github.com/Vladkryvoruchko/PSPNet-Keras-tensorflow/issues/63 https://github.com/Vladkryvoruchko/PSPNet-Keras-tensorflow/issues/49

As discussed in that issues .h5 and .json models generated by Keras(python3.5), are not supported in python3.6 This is a very general issue, that can occur again on some other versions, even if solution for this case will be provided.

• [ ] Put .npy weights on my personal DropBox: https://github.com/Vladkryvoruchko/PSPNet-Keras-tensorflow/issues/45
• [ ] Download .npy weights on the first run and convert them with current version of Python and Keras.
• [ ] Load only .npy weights and converting to .h5 .json automatically must resolve https://github.com/Vladkryvoruchko/PSPNet-Keras-tensorflow/issues/56
• [ ] Update to a newer version of TensorFlow/Keras/Scipy and resolve all incompatibilities with different Python versions starting from Python3.4+. Define best combinations if needed
• [ ] Define better environment in requirements.txt/README.md: https://github.com/Vladkryvoruchko/PSPNet-Keras-tensorflow/issues/38

I used pspnet101_voc2012.h5 and pspnet101_voc2012.json to test VOC data，and it's mIOU is 82.51% after uploaded the results. But the real value is 85.4% in the paper. I want to know why.

This is my resutls:

Competition | Category | Score -- | -- | -- Object Segmentation (comp6) | background | 95.69300 Object Segmentation (comp6) | aeroplane | 93.16350 Object Segmentation (comp6) | bicycle | 68.16070 Object Segmentation (comp6) | bird | 92.60880 Object Segmentation (comp6) | boat | 77.27700 Object Segmentation (comp6) | bottle | 82.46240 Object Segmentation (comp6) | bus | 91.86820 Object Segmentation (comp6) | car | 91.61700 Object Segmentation (comp6) | cat | 94.20940 Object Segmentation (comp6) | chair | 37.09920 Object Segmentation (comp6) | cow | 88.46340 Object Segmentation (comp6) | diningtable | 74.82890 Object Segmentation (comp6) | dog | 87.07190 Object Segmentation (comp6) | horse | 93.95220 Object Segmentation (comp6) | motorbike | 89.37420 Object Segmentation (comp6) | person | 90.35040 Object Segmentation (comp6) | pottedplant | 68.82600 Object Segmentation (comp6) | sheep | 90.44340 Object Segmentation (comp6) | sofa | 60.11600 Object Segmentation (comp6) | train | 85.58210 Object Segmentation (comp6) | tvmonitor | 79.56540   | Object Segmentation (comp6) | 82.51110

at my end when i run the command

python pspnet.py -m pspnet101_cityscapes -i example_images/cityscapes.png -o example_results/cityscapes.jpg

It always runs .npy weights by default. How to solve this issue? i have .h5 weights downloaded.