I'm implementing a couple new fcn networks for testing purposes, but for some reason I'm getting dimension errors when I call tf.train.AdamOptimizer in densenet_fcn.py on the model.

Here is the error:

 tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties:
name: GeForce GTX 1080
major: 6 minor: 1 memoryClockRate (GHz) 1.7335
pciBusID 0000:02:00.0
Total memory: 7.92GiB
Free memory: 256.12MiB
I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0
I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0:   Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080, pci bus id: 0000:02:00.0)
creating densenet model
(?, 384, 384, 21)
Running with tf training, initializing batches...
Traceback (most recent call last):
  File "densenet_fcn.py", line 132, in <module>
    train_step = tf.train.AdamOptimizer(learning_rate=0.0001).minimize(cross_entropy_sum)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/optimizer.py", line 269, in minimize
    grad_loss=grad_loss)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/optimizer.py", line 335, in compute_gradients
    colocate_gradients_with_ops=colocate_gradients_with_ops)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gradients_impl.py", line 482, in gradients
    in_grads = grad_fn(op, *out_grads)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/array_grad.py", line 348, in _ReshapeGrad
    return [array_ops.reshape(grad, array_ops.shape(op.inputs[0])), None]
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_array_ops.py", line 2448, in reshape
    name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 759, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2242, in create_op
    set_shapes_for_outputs(ret)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1617, in set_shapes_for_outputs
    shapes = shape_func(op)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1568, in call_with_requiring
    return call_cpp_shape_fn(op, require_shape_fn=True)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/common_shapes.py", line 610, in call_cpp_shape_fn
    debug_python_shape_fn, require_shape_fn)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/common_shapes.py", line 675, in _call_cpp_shape_fn_impl
    raise ValueError(err.message)
ValueError: Dimension size must be evenly divisible by 3096576 but is 147456 for 'adam_vars/gradients/Reshape_27_grad/Reshape' (op: 'Reshape') with input shapes: [384,384], [4].

have you seen anything like it or have advice on fixing the dimensions?

Daniil, your tensorFlow project is great for search human on images. Thanks for it! Let me introduce myself - Artyom, russian developer. And i want to ask you: is it possible to create tensorflow realisation with realtime detection from video? (like your cut a human from photo). Even on powerfull servers one image processing too slow for realtime project.

Is semantic segmentation not possible with this repository and upstream tensorflow and tensorflow/models alone as detailed in your comment?

It seems that, for example, upstream tensorflow/models vgg doesn't contain the parameter fc_conv_padding.

If specific forks/commits are needed, which ones should I use?

When importing the trained model, I got an error of .ckpt file not found. And actually the trained model seems not on the github.

May I ask where coud we download the trained model?

saver = tf.train.import_meta_graph('./fcn_8s_checkpoint/model_fcn8s_final.ckpt.meta') saver.restore(sess,"./fcn_8s_checkpoint/model_fcn8s_final.ckpt")

Got an error on TF 1.3.0 without GPU, any idea?

FailedPreconditionError Traceback (most recent call last) in () 60 saver.restore(sess,"./fcn_8s_checkpoint/model_fcn8s_final.ckpt") 61 ---> 62 image_np, pred_np = sess.run([image_tensor, pred], feed_dict=feed_dict_to_use) 63 64 io.imshow(image_np)

/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.pyc in run(self, fetches, feed_dict, options, run_metadata) 893 try: 894 result = self._run(None, fetches, feed_dict, options_ptr, --> 895 run_metadata_ptr) 896 if run_metadata: 897 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.pyc in _run(self, handle, fetches, feed_dict, options, run_metadata) 1122 if final_fetches or final_targets or (handle and feed_dict_tensor): 1123 results = self._do_run(handle, final_targets, final_fetches, -> 1124 feed_dict_tensor, options, run_metadata) 1125 else: 1126 results = []

/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.pyc in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata) 1319 if handle is None: 1320 return self._do_call(_run_fn, self._session, feeds, fetches, targets, -> 1321 options, run_metadata) 1322 else: 1323 return self._do_call(_prun_fn, self._session, handle, feeds, fetches)

/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.pyc in _do_call(self, fn, *args) 1338 except KeyError: 1339 pass -> 1340 raise type(e)(node_def, op, message) 1341 1342 def _extend_graph(self):

FailedPreconditionError: Attempting to use uninitialized value fcn_8s/pool3_fc/biases [[Node: fcn_8s/pool3_fc/biases/read = IdentityT=DT_FLOAT, _class=["loc:@fcn_8s/pool3_fc/biases"], _device="/job:localhost/replica:0/task:0/cpu:0"]]

Caused by op u'fcn_8s/pool3_fc/biases/read', defined at: File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main "main", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/usr/local/lib/python2.7/dist-packages/ipykernel_launcher.py", line 16, in app.launch_new_instance() File "/usr/local/lib/python2.7/dist-packages/traitlets/config/application.py", line 658, in launch_instance app.start() File "/usr/local/lib/python2.7/dist-packages/ipykernel/kernelapp.py", line 477, in start ioloop.IOLoop.instance().start() File "/usr/local/lib/python2.7/dist-packages/zmq/eventloop/ioloop.py", line 177, in start super(ZMQIOLoop, self).start() File "/usr/local/lib/python2.7/dist-packages/tornado/ioloop.py", line 888, in start handler_func(fd_obj, events) File "/usr/local/lib/python2.7/dist-packages/tornado/stack_context.py", line 277, in null_wrapper return fn(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/zmq/eventloop/zmqstream.py", line 440, in _handle_events self._handle_recv() File "/usr/local/lib/python2.7/dist-packages/zmq/eventloop/zmqstream.py", line 472, in _handle_recv self._run_callback(callback, msg) File "/usr/local/lib/python2.7/dist-packages/zmq/eventloop/zmqstream.py", line 414, in _run_callback callback(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/tornado/stack_context.py", line 277, in null_wrapper return fn(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/ipykernel/kernelbase.py", line 283, in dispatcher return self.dispatch_shell(stream, msg) File "/usr/local/lib/python2.7/dist-packages/ipykernel/kernelbase.py", line 235, in dispatch_shell handler(stream, idents, msg) File "/usr/local/lib/python2.7/dist-packages/ipykernel/kernelbase.py", line 399, in execute_request user_expressions, allow_stdin) File "/usr/local/lib/python2.7/dist-packages/ipykernel/ipkernel.py", line 196, in do_execute res = shell.run_cell(code, store_history=store_history, silent=silent) File "/usr/local/lib/python2.7/dist-packages/ipykernel/zmqshell.py", line 533, in run_cell return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", line 2718, in run_cell interactivity=interactivity, compiler=compiler, result=result) File "/usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", line 2822, in run_ast_nodes if self.run_code(code, result): File "/usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", line 2882, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "", line 49, in is_training=False) File "tf_image_segmentation/utils/inference.py", line 53, in new_network_definition all_outputs = network_definition(*args, **kwargs) File "tf_image_segmentation/models/fcn_8s.py", line 151, in FCN_8s scope='pool3_fc') File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 181, in func_with_args return func(*args, **current_args) File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/layers/python/layers/layers.py", line 1027, in convolution outputs = layer.apply(inputs) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/layers/base.py", line 503, in apply return self.call(inputs, *args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/layers/base.py", line 443, in call self.build(input_shapes[0]) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/layers/convolutional.py", line 144, in build dtype=self.dtype) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/layers/base.py", line 383, in add_variable trainable=trainable and self.trainable) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variable_scope.py", line 1065, in get_variable use_resource=use_resource, custom_getter=custom_getter) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variable_scope.py", line 962, in get_variable use_resource=use_resource, custom_getter=custom_getter) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variable_scope.py", line 360, in get_variable validate_shape=validate_shape, use_resource=use_resource) File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/layers/python/layers/layers.py", line 1561, in layer_variable_getter return _model_variable_getter(getter, *args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/layers/python/layers/layers.py", line 1553, in _model_variable_getter custom_getter=getter, use_resource=use_resource) File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 181, in func_with_args return func(*args, **current_args) File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/framework/python/ops/variables.py", line 261, in model_variable use_resource=use_resource) File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 181, in func_with_args return func(*args, **current_args) File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/framework/python/ops/variables.py", line 216, in variable use_resource=use_resource) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variable_scope.py", line 352, in _true_getter use_resource=use_resource) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variable_scope.py", line 725, in _get_single_variable validate_shape=validate_shape) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variables.py", line 199, in init expected_shape=expected_shape) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variables.py", line 330, in _init_from_args self._snapshot = array_ops.identity(self._variable, name="read") File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_array_ops.py", line 1400, in identity result = _op_def_lib.apply_op("Identity", input=input, name=name) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op op_def=op_def) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2630, in create_op original_op=self._default_original_op, op_def=op_def) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1204, in init self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

FailedPreconditionError (see above for traceback): Attempting to use uninitialized value fcn_8s/pool3_fc/biases [[Node: fcn_8s/pool3_fc/biases/read = IdentityT=DT_FLOAT, _class=["loc:@fcn_8s/pool3_fc/biases"], _device="/job:localhost/replica:0/task:0/cpu:0"]]

If you're running resnet with the latest tensorflow and tensorflow/models you will need the following: https://github.com/tensorflow/models/pull/1447

I plan to update this with additional info for running on tensorflow 1.1

You mentioned there is a more efficient function for converting the colored segmentation pngs to gray class pngs where the value is the class number. Which is the preferred function to do this?

while training I am getting error in line valid_logits_batch_tensor = tf.gather_nd(params=logits_batch_tensor, indices=valid_batch_indices) in function training.py

Error:

InvalidArgumentError (see above for traceback): Invalid indices: [31,1] = [0, 76, 192] does not index into [1,192,192,21] [[Node: adam_vars/gradients/GatherNd_1_grad/ScatterNd = ScatterNd[T=DT_FLOAT, Tindices=DT_INT32, _device="/job:localhost/replica:0/task:0/cpu:0"](ToInt32_3/_587, adam_vars/gradients/Reshape_4_grad/Reshape/_589, adam_vars/gradients/GatherNd_1_grad/Shape/_591)]] [[Node: adam_vars/gradients/GatherNd_1_grad/ScatterNd/_593 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_252_adam_vars/gradients/GatherNd_1_grad/ScatterNd", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]]

Training code

Did any of you faced similar issue ?

using convert_pascal_voc_to_tfrecords.ipynb to convert pascal and pascal_berkeley in to tfrecord and got a error: Traceback (most recent call last): File "convert_pascal_voc_to_tfrecords.ipynb", line 18, in mode=2) File "/home/cxy/Desktop/FCN_MIL_supervision_tensorflow-master/code/tf_image_segmentation/utils/pascal_voc.py", line 557, in get_augmented_pascal_image_annotation_filename_pairs train_from_berkeley_image_annotation_pairs + train_from_pascal_image_annotation_pairs TypeError: unsupported operand type(s) for +: 'zip' and 'zip'

Did anyone try to create a .pb file(protocol buffer) for this model? I have created and ran .pb file for other models but I am getting errors with this one. I am facing some issues while feeding the input to the .pb file.

Hi,

How can I modify the code to be used for semantic segmentation of video frames. I used open-cv to process the frames of the video and basically send each frame to the model for segmentation but I'm getting an error 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte. How do I fix this?

I am trying to run pertained model with a test image. after settling lot of things still I got certain error like

InvalidArgumentError (see above for traceback): Incompatible shapes: [1,28,28,21] vs. [1,16,16,21]

[[Node: fcn_8s/add = Add[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](fcn_8s/pool4_fc/BiasAdd, fcn_8s/conv2d_transpose)]]

Do I need to change the shape of the image first? But for each test image it says different dimensions vs different dimensions

Moreover whats the purpose of num_classes in pertained model? and what should it be for a test image?

I am looking for a tutorial or step-by-step instructions including pre-processing, labeling and creation of tfrecords for building an image segmentation pipeline. Any recommendations, especially on the labeling of a multi-class image and converting to tfrecords correct would be great! Thank you!

tjmac

When runing the following code I get this error "InvalidArgumentError (see above for traceback): logits and labels must be same size: logits_size=[1,65536] labels_size=[65536,1]" in cross_entropy_with_logits. I can work around this by doing

cross_entropies = tf.nn.softmax_cross_entropy_with_logits(logits=valid_logits_batch_tensor,
                                                          labels=tf.transpose(valid_labels_batch_tensor))

but not

cross_entropies = tf.nn.softmax_cross_entropy_with_logits(logits=tf.transpose(valid_logits_batch_tensor),
                                                          labels=valid_labels_batch_tensor)

And then when training the loss only increases continuously. I have changed the learning rate couple of times to no avail. I'm using images with 2 classes (Background and Ground Truth)

import numpy as np
import tensorflow as tf
import skimage.io as io
#import tensorflow.contrib.slim.nets

from matplotlib import pyplot as plt
from nets import vgg
from preprocessing import vgg_preprocessing

# Load the mean pixel values and the function
# that performs the subtraction from each pixel
from preprocessing.vgg_preprocessing import (_mean_image_subtraction,
                                            _R_MEAN, _G_MEAN, _B_MEAN)
from tf_image_segmentation.utils.tf_records import read_tfrecord_and_decode_into_image_annotation_pair_tensors
from tf_image_segmentation.models.fcn_32s import FCN_32s, extract_vgg_16_mapping_without_fc8

from tf_image_segmentation.utils.pascal_voc import pascal_segmentation_lut

from tf_image_segmentation.utils.training import get_valid_logits_and_labels

from tf_image_segmentation.utils.augmentation import (distort_randomly_image_color,
                                                      flip_randomly_left_right_image_with_annotation,
                                                      scale_randomly_image_with_annotation_with_fixed_size_output)

#os.environ["CUDA_VISIBLE_DEVICES"] = '0'




def my_detection_lut():
    """Return look-up table with number and correspondng class names
    for PASCAL VOC segmentation dataset. Two special classes are: 0 -
    background and 255 - ambigious region. All others are numerated from
    1 to 20.
    
    Returns
    -------
    classes_lut : dict
        look-up table with number and correspondng class names
    """

    class_names = ['background', 'GT', 'ambigious']
    
    enumerated_array = enumerate(class_names[:-1])
    
    classes_lut = list(enumerated_array)
    
    # Add a special class representing ambigious regions
    # which has index 255.
    classes_lut.append((255, class_names[-1]))
    
    classes_lut = dict(classes_lut)

    return classes_lut
root_folder = os.getcwd()
img_folder = os.path.join(root_folder, 'data\\images')
checkpoints_dir = os.path.join(root_folder, 'data')
vgg_checkpoint_path = os.path.join(checkpoints_dir, 'vgg_16.ckpt')
log_folder = os.path.join(root_folder, 'logs')
model_ckpt_path = os.path.join(root_folder, 'model_fcn32.ckpt')
slim = tf.contrib.slim


dataset_len = len([name for name in os.listdir(os.path.join(img_folder,'processed\\frames'))])
print(dataset_len)
trainset_len = int(dataset_len * 0.25)
testset_len = dataset_len - trainset_len
image_train_size = [256,256]
number_of_classes = 2
tfrecord_filename = 'polyp_test_pairs.tfrecords'
tfrecord_filename = os.path.join(img_folder, 'polyp_test_pairs.tfrecords')
print(tfrecord_filename)
my_detection = my_detection_lut()
class_labels = list(my_detection.keys())


tf.reset_default_graph() 

filename_queue = tf.train.string_input_producer([tfrecord_filename], num_epochs=10)
image, annotation = read_tfrecord_and_decode_into_image_annotation_pair_tensors(filename_queue)

resized_image = tf.image.resize_images(images=image,
                                           size=image_train_size)
    
resized_annotation = tf.image.resize_images(images=annotation,
                                           size=image_train_size)


image_batch, annotation_batch = tf.train.shuffle_batch([resized_image, resized_annotation],
                                                       batch_size=1, capacity=20, num_threads=1,
                                                       min_after_dequeue=10)


upsampled_logits_batch, vgg_16_variables_mapping = FCN_32s(image_batch_tensor=image_batch,
                                                           number_of_classes=number_of_classes,
                                                           is_training=True)

valid_labels_batch_tensor, valid_logits_batch_tensor = get_valid_logits_and_labels(annotation_batch_tensor=annotation_batch,
                                                                                     logits_batch_tensor=upsampled_logits_batch,
                                                                                    class_labels=class_labels)

cross_entropies = tf.nn.softmax_cross_entropy_with_logits(logits=valid_logits_batch_tensor,
                                                          labels=valid_labels_batch_tensor)

# Normalize the cross entropy -- the number of elements
# is different during each step due to mask out regions
cross_entropy_sum = tf.reduce_mean(cross_entropies)

pred = tf.argmax(upsampled_logits_batch, dimension=3)

probabilities = tf.nn.softmax(upsampled_logits_batch)


with tf.variable_scope("adam_vars"):
    train_step = tf.train.AdamOptimizer(learning_rate=0.000001).minimize(cross_entropy_sum)


# Variable's initialization functions
vgg_16_without_fc8_variables_mapping = extract_vgg_16_mapping_without_fc8(vgg_16_variables_mapping)


init_fn = slim.assign_from_checkpoint_fn(model_path=vgg_checkpoint_path,
                                         var_list=vgg_16_without_fc8_variables_mapping)

global_vars_init_op = tf.global_variables_initializer()

tf.summary.scalar('cross_entropy_loss', cross_entropy_sum)

merged_summary_op = tf.summary.merge_all()

summary_string_writer = tf.summary.FileWriter(log_folder)

# Create the log folder if doesn't exist yet
if not os.path.exists(log_folder):
     os.makedirs(log_folder)
    
#The op for initializing the variables.
local_vars_init_op = tf.local_variables_initializer()

combined_op = tf.group(local_vars_init_op, global_vars_init_op)

# We need this to save only model variables and omit optimization-related and other variables.
model_variables = slim.get_model_variables()
saver = tf.train.Saver(model_variables)

with tf.Session()  as sess:
    
    sess.run(combined_op)
    init_fn(sess)
    summary_string_writer.add_graph(sess.graph)

    coord = tf.train.Coordinator()
    threads = tf.train.start_queue_runners(coord=coord)

    for i in range(testset_len*10):
        cross_entropy, summary_string, _ = sess.run([ cross_entropy_sum,
                                                      merged_summary_op,
                                                      train_step ])
        print(i,") Current loss: " + str(cross_entropy))
        summary_string_writer.add_summary(summary_string, i)           
        if i % trainset_len == 0:
            save_path = saver.save(sess, model_ckpt_path)
            print("Model saved in file: %s" % save_path)
    coord.request_stop()
    coord.join(threads)            
    save_path = saver.save(sess, model_ckpt_path)
    print("Model saved in file: %s" % save_path)
summary_string_writer.close()

你好，我现在在运行这个代码时产生了下面的错误 Hello, I now have the following error when I run data_pascal_voc.py Using TensorFlow backend. Usage: G:/python code/kreas-FCN-ATH/src/tf-image-segmentation-ahundt-keras/tf_image_segmentation/recipes/pascal_voc/data_pascal_voc.py [(with UPDATE...)] [options] G:/python code/kreas-FCN-ATH/src/tf-image-segmentation-ahundt-keras/tf_image_segmentation/recipes/pascal_voc/data_pascal_voc.py help [COMMAND] G:/python code/kreas-FCN-ATH/src/tf-image-segmentation-ahundt-keras/tf_image_segmentation/recipes/pascal_voc/data_pascal_voc.py (-h | --help) G:/python code/kreas-FCN-ATH/src/tf-image-segmentation-ahundt-keras/tf_image_segmentation/recipes/pascal_voc/data_pascal_voc.py COMMAND [(with UPDATE...)] [options] Closing remaining open files:vgg16_weights_tf_dim_ordering_tf_kernels.h5...donevgg16_weights_tf_dim_ordering_tf_kernels_notop.h5...done I do not understand why, I run under windows, I hope to get your answer, thank you.