When I train voc data, the error happened. My GPU is RTX2080 8G * 2，tensorflow-gpu:1.12，keras2.2.4

Epoch 1/50 2019-01-28 00:16:00.441512: E tensorflow/stream_executor/cuda/cuda_blas.cc:652] failed to run cuBLAS routine cublasSgemm_v2: CUBLAS_STATUS_EXECUTION_FAILED Traceback (most recent call last): File "train.py", line 192, in <module> _main(annotation_path=anno) File "train.py", line 65, in _main callbacks=[logging, checkpoint]) File "/usr/local/lib/python3.6/dist-packages/keras/legacy/interfaces.py", line 91, in wrapper return func(*args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 1418, in fit_generator initial_epoch=initial_epoch) File "/usr/local/lib/python3.6/dist-packages/keras/engine/training_generator.py", line 217, in fit_generator class_weight=class_weight) File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 1217, in train_on_batch outputs = self.train_function(ins) File "/usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py", line 2715, in __call__ return self._call(inputs) File "/usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py", line 2675, in _call fetched = self._callable_fn(*array_vals) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1439, in __call__ run_metadata_ptr) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/errors_impl.py", line 528, in __exit__ c_api.TF_GetCode(self.status.status)) tensorflow.python.framework.errors_impl.InternalError: Blas SGEMM launch failed : m=346112, n=32, k=64 [[{{node conv2d_3/convolution}} = Conv2D[T=DT_FLOAT, _class=["loc:@batch_normalization_3/cond/FusedBatchNorm/Switch"], data_format="NHWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](leaky_re_lu_2/LeakyRelu, conv2d_3/kernel/read)]] [[{{node yolo_loss/while_1/LoopCond/_2963}} = _HostRecv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_6607_yolo_loss/while_1/LoopCond", tensor_type=DT_BOOL, _device="/job:localhost/replica:0/task:0/device:CPU:0"](^_cloopyolo_loss/while_1/strided_slice_1/stack_2/_2805)]]

When I save weights during training using your original code, I got:

InvalidArgumentError (see above for traceback): Input to reshape is a tensor with 12675 values, but the requested shape requires a multiple of 3042
	 [[Node: Reshape_3 = Reshape[T=DT_FLOAT, Tshape=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"](conv2d_59/BiasAdd, Reshape_3/shape)]]

Does this mean I should save model instead of weights? May I ask why in your code "model.save_weights" is used instead of "model.save_path"?

When I saved checkpoints in train.py as keras model, I wasn't able to use it in yolo.py using load_model.

checkpoint = ModelCheckpoint(log_dir + "checkpoint.h5", monitor='val_loss', save_best_only=True)

history = model.fit([image_data, *y_true],
          np.zeros(len(image_data)),
          validation_split=.1,
          batch_size=Batch_Size,
          epochs=10000,
          callbacks=[checkpoint])

When I use yolo.py to test the trained model on images, model is loaded using

self.yolo_model = load_model(model_path, compile=False)

I got this error:

2018-05-10 05:47:25.060203: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
Traceback (most recent call last):
  File "/home/jin/workspace/Intersection_TrafficFlow/detection/keras-yolo3-noaug/keras-yolo3-1/yolo.py", line 229, in <module>
    detect_img(YOLO())
  File "/home/jin/workspace/Intersection_TrafficFlow/detection/keras-yolo3-noaug/keras-yolo3-1/yolo.py", line 58, in __init__
    self.boxes, self.scores, self.classes = self.generate()
  File "/home/jin/workspace/Intersection_TrafficFlow/detection/keras-yolo3-noaug/keras-yolo3-1/yolo.py", line 80, in generate
    self.yolo_model = load_model(model_path, compile=False)
  File "/home/jin/anaconda3/envs/keras-yolo3-noaug/lib/python3.6/site-packages/keras/models.py", line 243, in load_model
    model = model_from_config(model_config, custom_objects=custom_objects)
  File "/home/jin/anaconda3/envs/keras-yolo3-noaug/lib/python3.6/site-packages/keras/models.py", line 317, in model_from_config
    return layer_module.deserialize(config, custom_objects=custom_objects)
  File "/home/jin/anaconda3/envs/keras-yolo3-noaug/lib/python3.6/site-packages/keras/layers/__init__.py", line 55, in deserialize
    printable_module_name='layer')
  File "/home/jin/anaconda3/envs/keras-yolo3-noaug/lib/python3.6/site-packages/keras/utils/generic_utils.py", line 144, in deserialize_keras_object
    list(custom_objects.items())))
  File "/home/jin/anaconda3/envs/keras-yolo3-noaug/lib/python3.6/site-packages/keras/engine/topology.py", line 2524, in from_config
    process_node(layer, node_data)
  File "/home/jin/anaconda3/envs/keras-yolo3-noaug/lib/python3.6/site-packages/keras/engine/topology.py", line 2483, in process_node
    layer(input_tensors, **kwargs)
  File "/home/jin/anaconda3/envs/keras-yolo3-noaug/lib/python3.6/site-packages/keras/engine/topology.py", line 619, in __call__
    output = self.call(inputs, **kwargs)
  File "/home/jin/anaconda3/envs/keras-yolo3-noaug/lib/python3.6/site-packages/keras/layers/core.py", line 663, in call
    return self.function(inputs, **arguments)
  File "/home/jinz1/Jin/Intersection_TrafficFlow/detection/keras-yolo3-noaug/keras-yolo3-1/yolo3/model.py", line 347, in yolo_loss
NameError: name 'yolo_head' is not defined

Thank you very much for your help and for creating this repo!

I get this error: minh@minh-Aspire-A515-51G:~/keras-yolo3$ python train.py /home/minh/.local/lib/python3.5/site-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. from ._conv import register_converters as _register_converters Using TensorFlow backend. Traceback (most recent call last): File "train.py", line 153, in _main() File "train.py", line 31, in _main input_shape, max_boxes=100, load_previous=True) File "train.py", line 76, in get_training_data image = Image.open(filename) File "/home/minh/.local/lib/python3.5/site-packages/PIL/Image.py", line 2548, in open fp = builtins.open(filename, "rb") FileNotFoundError: [Errno 2] No such file or directory: '000012\n'

I don't know where is the file 'train.txt' and i got this file in VOCdevkit/VOC2007/ImageSets/Main? Can you help me?

I added in train.py a model.save(log_dir + 'model.h5') next to the model.save_weights statement. I use the method gived by amai to convert the model.h5 to model.pb file, but I get the error that yolo_head is not defined in yolo_loss.

when I use the "load_model()", I get the following error:

from keras.models import load_model,Model
base_model = load_model('yolo_model.h5') #yolo_model.h5 is the model trained on my own 
                                                                       #dataset.
print(base_model.summary())

.... File "C:\Users\ww\Desktop\PycharmProjects\YOLO\keras-yolo3\yolo3\model.py", line 459, in yolo_loss grid, raw_pred, pred_xy, pred_wh = yolo_head(yolo_outputs[l], NameError: name 'yolo_head' is not defined

I used my own data to train the model detect the specific class, and the model run well in darknet. But when I try to convert the model , I can not detect any object. I use my own cfg and my own .weight file when convert the model, and I change the path of model path and class path. Is there any place need to change?

I want to use the code to detect the ID card. Right now I have some data collected and labeled. Is it OK I run train.py with only my dataset, which only has 1 class: id card?

My error occur when i use model tiny yolo? Error in yolo.py

---> 35         self.boxes, self.scores, self.classes = self.generate()
     36 
     37     def _get_class(self):

~/Documents/Object_Detection/keras-yolo3/yolo.py in generate(self)
     66             assert self.yolo_model.layers[-1].output_shape[-1] == \
     67                 num_anchors/len(self.yolo_model.output) * (num_classes + 5), \
---> 68                 'Mismatch between model and given anchor and class sizes'
     69 
     70         print('{} model, anchors, and classes loaded.'.format(model_path))

AssertionError: Mismatch between model and given anchor and class sizes

I'm trying to use yolo3 with tensorflow serving and I'm having hard time converting returned response values.

I exported Keras model for TF Serving and as I can see there are three outputs available in keras model as follow:

>> yolo_keras_model.output
[<tf.Tensor 'conv2d_59_1/BiasAdd:0' shape=(?, ?, ?, 255) dtype=float32>,
 <tf.Tensor 'conv2d_67_1/BiasAdd:0' shape=(?, ?, ?, 255) dtype=float32>,
 <tf.Tensor 'conv2d_75_1/BiasAdd:0' shape=(?, ?, ?, 255) dtype=float32>]

... I exported the model with following signature:

signature = predict_signature_def(
    inputs={'inputs': yolo_keras_model.input}, 
    outputs={
            'conv2d_59': yolo_keras_model.output[0],
            'conv2d_67': yolo_keras_model.output[1],
            'conv2d_75': yolo_keras_model.output[2]
    }

I'm getting following response in tensroflow serving client script:

response = stub.Predict(request, 60.0)
conv2d_59 = np.squeeze(tf.contrib.util.make_ndarray(response.outputs['conv2d_59']))
conv2d_67 = np.squeeze(tf.contrib.util.make_ndarray(response.outputs['conv2d_67']))
conv2d_75 = np.squeeze(tf.contrib.util.make_ndarray(response.outputs['conv2d_67']))

>> conv2d_59.shape, conv2d_67.shape, conv2d_75.shape
((15, 25, 255), (30, 50, 255), (60, 100, 255))

The confusion here is, how should I interpret each outputs? Are these values correlate with box, score, classes values?

Hi,

I see the yolo head in model.py line 140

feats[...,:2] is after sigmoid function https://github.com/qqwweee/keras-yolo3/blob/e6598d13c703029b2686bc2eb8d5c09badf42992/yolo3/model.py#L140

But in the loss function raw_true_xy is just inverse of line 140 without sigmoid https://github.com/qqwweee/keras-yolo3/blob/e6598d13c703029b2686bc2eb8d5c09badf42992/yolo3/model.py#L380 ... In xy_loss ,raw_pred[...,0:2] also didn't have sigmoid function https://github.com/qqwweee/keras-yolo3/blob/e6598d13c703029b2686bc2eb8d5c09badf42992/yolo3/model.py#L399

Why in yolo_head need sigmoid but train didn't need

Did i miss something?

Thanks!

hello, when i run python train.py, it returns error as: type error,load_weights() got 2 argument 'skip_mismatch' argument error, it seems like 'skip_mismatch'returns the argument error,whats the problem, my environment setting as keras version:2.1.5, pls help,tks,by simon

1. Suppose there's a picture with size (1200,800) and objectA is in (x,y,w,h)=(300,300,200,100), do I have to change the ratio to (416,416) or just input (300,300,200,100)?

2. whether the output of (x,y,w,z,....) is in (416,416) or the original size (1200,800) ?

thanks

---> 66 y_true[l][b, j, i, k, 0:4] = true_boxes[b,t, 0:4] 67 y_true[l][b, j, i, k, 4] = 1 68 y_true[l][b, j, i, k, 5+c] = 1

IndexError: index 13 is out of bounds for axis 2 with size 13

Hey guys, I've run all the quick start commands and I'm now at the training section Am I to just,

1. Run python voc_annotation.py (I have no custom datasets)
2. And then run python convert.py -w yolov3.cfg yolov3.weights model_data/yolo_weights.h5
3. And then finally run train.py? Is this going to work please?

Following is the link to the exact line of code. I just want to understand the theory behind it. https://github.com/qqwweee/keras-yolo3/blob/e6598d13c703029b2686bc2eb8d5c09badf42992/train.py#L182