Given our manpower (spoiler alert, it's not much), we need to choose carefully how we select features which will stay in the repo.

Given that this repo will host a lot of features (hopefully more than in keras!) the maintainers cannot do everything. Which is why each layer/optimizer/callback/...etc should be assigned to a contributor or a maintainer who will commit to answer issues and review pull requests about it. The assignee will also commit to have proper unit tests for his/her feature and proper docstrings.

The maintainers from the keras team will be in charge of the general repo health and coding standards.

About the things that I will commit to maintain myself:

• The Swish activation
• The TensorBoardGrouped callback
• The GroupNormalization layer

If you care about one of the features which is in this repo and think that you can take care of issues, PRs, unit tests and docstrings related to it, then feel free to drop a message here.

All features without someone assigned to it will be removed in v1.0.

Contributions w.r.t. the following layers and examples are welcomed:

• Layers
  • Layer Normalization
  • PELU [DONE!]
  • Batch Renormalization
• Examples
  • DenseNets [DONE!]
  • Residual of Residual Networks [DONE!]
  • Wide ResNets[DONE!]
  • ResNet in ResNet

I have added a Capsule Layer implementation (pure Keras), referring to this example on the Keras repository. Here is the related issue on the Keras repository.

About TFRecords: https://www.tensorflow.org/api_guides/python/python_io Code Source: https://github.com/indraforyou/keras_tfrecord Primary Author: Indranil Sur @indraforyou License: MIT (same as Keras)

The Rectified Linear Unit with Sigmoid has been under development for almost a year now. It has been part of the IBM Watson AI XPRIZE Competition and benchmarked against the ReLU and Leaky ReLU, and still showing better results.

The following data sets were used:

• MNIST
• IMDB
• Kaggle Toxicity

The function will be demonstrated publicly at the Codemotion Amsterdam 2018 Conference and the DevDays in Vilnius, Lithuania. Both conferences are happening in May this year.

Jupyter notebooks using the function are available here: https://github.com/ekholabs/DLinK

Below, some benchmarking against ReLU and Leaky ReLU using Deep Nets and CNNs.

• Deep Net

• CNN

The function has also showed better results with DQN for the CartPole.v0 and ResNet 50.

Since I'm not linked to any academic institution, it has been proven difficult to make the function open. However, as a follower of the open source movement, I think it should be made available for others to test / use. I hope the Keras community can find a good use for it.

I set up a BILSTM-CRF model for sequence labelling, very similar to this example (https://github.com/farizrahman4u/keras-contrib/blob/master/examples/conll2000_chunking_crf.py).

from keras.layers  import Dense, Masking
from keras.layers import Dropout
from keras.layers import Bidirectional
from keras.layers import LSTM
from keras.models import Sequential
from keras.optimizers import Adam
from keras_contrib.layers import CRF

self._model = Sequential(name='core_sequential')
self._model.add(Masking(mask_value=0., input_shape=(None, input_size)))

self._model.add(Dropout(dropout_rate,name='dropout_layer_1'))
self._model.add(Bidirectional(LSTM(units=hidden_unit_size,
                                     return_sequences=True,
                                     activation="tanh", 
                                     name="lstm_layer"),
                                     name='birnn_layer'))
crf = CRF(2, sparse_target=True)
self._model.add(crf)
self._model.compile(optimizer=Adam(lr=lr), loss=crf.loss_function, metrics=[crf.accuracy])

self._model.fit(x_train, y_train, validation_data=(x_val, y_val), epochs=epochs, batch_size=batch_size,
                            verbose=verbose, shuffle=shuffle)

self._model.save(filename)

The model was trained successfully and I also successfully called the predict_classes() function, as soon as I trained it and I had the object after training...

Then I tried to load it back, like it is mentioned in the wiki "A Common "Gotcha", importing the additional layer, before calling load_model() function :

from keras.models import load_model
from keras_contrib.layers import CRF

self._model = load_model(filename)

The error message is:

[...]
  File "/Users/kiddo/anaconda/lib/python3.6/site-packages/keras/utils/generic_utils.py", line 140, in deserialize_keras_object
    list(custom_objects.items())))
  File "/Users/kiddo/anaconda/lib/python3.6/site-packages/keras/models.py", line 1202, in from_config
    layer = layer_module.deserialize(conf, custom_objects=custom_objects)
  File "/Users/kiddo/anaconda/lib/python3.6/site-packages/keras/layers/__init__.py", line 54, in deserialize
    printable_module_name='layer')
  File "/Users/kiddo/anaconda/lib/python3.6/site-packages/keras/utils/generic_utils.py", line 133, in deserialize_keras_object
    ': ' + class_name)
**ValueError: Unknown layer: CRF**

So, I changed to:

from keras_contrib.layers import CRF

self._model = load_model(filename, custom_objects={'CRF':CRF})

The new error says that the loss is missing also:

  File "/Users/kiddo/anaconda/lib/python3.6/site-packages/keras/engine/training.py", line 751, in compile
    loss_function = losses.get(loss)
  File "/Users/kiddo/anaconda/lib/python3.6/site-packages/keras/losses.py", line 96, in get
    return deserialize(identifier)
  File "/Users/kiddo/anaconda/lib/python3.6/site-packages/keras/losses.py", line 88, in deserialize
    printable_module_name='loss function')
  File "/Users/kiddo/anaconda/lib/python3.6/site-packages/keras/utils/generic_utils.py", line 157, in deserialize_keras_object
    ':' + function_name)
**ValueError: ('Unknown loss function', ':loss')**

I tried extending the custom_objects dict with ':loss':CRF.CRF.loss_function', but I still have the same error...

Any idea about that?

Adds DenseNet FCN model builder script, depth_to_space operations in backend and SubpixelConvolutionUpsampling layer.

Todo:

1. See if we can test depth_to_space (I don't know how to do this). But I have used the Subpixel convolutional layer for upsampling tasks which uses this, and output is correct. Does it imply that the operations in depth_to_space is correct as well?

2. See if we can somehow test SubpixelUsampling layer. The problem is it does not take in an input size, it simply receives the output of the previous layer and shuffles it to obtain a larger size in exchange for smaller number of kernels. Implementation should be correct, but tests need to be designed to work with an upsampling technique which is this unique.

Hi @fchollet and @ahundt,

I put some more time in the SineReLU and improved it, from a performance and running time perspective. The function is now simpler, faster and the results are better.

• Increase speed by reducing the computation: instead of using sigmoid and e, just use the difference of sin(Z) and cos(Z).
• Make the function differentiable

I wrote a Medium post about it, which will serve as inspiration for a paper to, hopefully, get out next year (which will be way more technical than the post).

Next to the documentation written in the code, I also took the time run it several times and get a more interesting comparison:

Link for the Medium story: https://medium.com/@wilder.rodrigues/sinerelu-an-alternative-to-the-relu-activation-function-e46a6199997d

Thanks in advance.

Callback implementation of a cyclical learning rate policy, as described in Leslie Smith's paper Cyclical Learning Rates for Training Neural Networks.

I'm training and testing DenseNetFCN on Pascal VOC2012

Could I get advice on next steps to take to debug and improve the results?

To do so I'm using the train.py training script in my fork of Keras-FCN in conjunction with DenseNetFCN in keras-contrib with #46 applied, which for DenseNetFCN really mostly changes the formatting for pep8, though the regular densenet is more heavily modified.

I use Keras-FCN because we don't have an FCN training script here in keras-contrib yet, though I plan to adapt & submit this here once things work properly. While the papers don't publish results on Pascal VOC, the original DenseNet has fairly state of the art results on ImageNet & cifar10/cifar100, and DenseNetFCN performed well on CamVid and the Gatech dataset. Considering this I expected that perhaps DenseNetFCN might fail to get state of the art results, but I figured it could most likely in a worst case get over 50% mIOU and around 70-80% pixel accuracy since it has many similarities in common with ResNet, and performed quite well on the much smaller CamVid dataset.

DenseNet FCN configuration I'm using

    return densenet.DenseNetFCN(input_shape=(320, 320, 3),
                                weights=None, classes=classes,
                                nb_layers_per_block=4,
                                growth_rate=13,
                                dropout_rate=0.2)

This is very close to the configuration FC-DenseNet56 from the 100 layers tiramisu aka DenseNetFCN paper.

Sparse training accuracy

Here is what I'm seeing as I train with the Adam optimizer and loss rate of 0.1:

lr: 0.100000
Epoch 2/450
366/366 [==============================] - 287s - loss: 62.5131 - 
sparse_accuracy_ignoring_last_label: 0.3657
[...snip...]
lr: 0.100000
Epoch 148/450
366/366 [==============================] - 286s - loss: 82.2138 - 
sparse_accuracy_ignoring_last_label: 0.3375

Similar networks and training scripts verified for a baseline

1. I've successfully trained AtrousFCNResNet50_16s from scratch up to around 7x% pixel accuracy on the pascal VOC2012 test set.
2. AtrousFCNResNet50_16s with resnet from the keras pretrained weights as fchollet provided downloaded using the Keras-FCN get_weights_path and transfer_FCN script trained without issue to 0.56025 mIOU and around 8x% accuracy.
3. I've been able to train plain DenseNet on cifar 10 up to expected levels of accuracy published in the original papers.
4. I've been able to train DenseNetFCN on a single image, and predict pixels with 99% accuracy on the image itself with all augmentation disabled, and mid 9x% accuracy on augmented versions of the training image.
   • This should at least demonstrate that the network can be trained, and the labels aren't saved incorrectly. I know (4) isn't a valid experiment for final conclusions, it just helps eliminate a variety of possible bugs.

Verifying training scripts with AtrousFCNResNet50_16s

For comparison, AtrousFCNResNet50_16s test set training results, which can be brought to 0.661 mIOU with augmented pascal voc,

I also trained AtrousFCNResNet50_16s from scratch with the results below:

PASCAL VOC trained with pretrained imagenet weights
IOU:
[ 0.90873648  0.74772504  0.44416247  0.57239141  0.50728778  0.51896323
  0.69891196  0.66111323  0.64380596  0.19145411  0.49733934  0.32720705
  0.5488089   0.49649298  0.6157158   0.75780816  0.35492963  0.57446371
  0.32721105  0.63200183  0.53067634]
meanIOU: 0.550343
pixel acc: 0.896132
150.996609926s used to calculate IOU.

Download links

Pascal VOC
http://host.robots.ox.ac.uk/pascal/VOC/voc2012/VOCtrainval_11-May-2012.tar  

Augmented 
http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/semantic_contours/benchmark.tgz

Automated VOC download script

Thanks

@titu1994 and @aurora95, since I'm using your respective DenseNetFCN and Keras-FCN implementations, could I get any comments or advice you might have on this? I'd appreciate your thoughts.

All, thanks for giving this a look, as well as your consideration and advice!

Adds Batch Renormalization layer from the paper Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models by Sergey Ioffe

SineReLU gone rough in Sequential Method after being called through string alias method

Previously, I had called SineReLU function in subclass model building method using string alias through get_custom_objects(), Unfortunately, BatchNormalization function failed in that method producing an interesting error of invalid tesnor rank.

Don't want to be carried out! I then swtiched to something called Sequential method, yada! Our favourite proven way!! But, this time, it shows a typesec error with the SineReLU function which is totally weird. Because, I have improted and run the functional call in previous cell and all works perfectly fine except while I call the function within the core model architecture.

I am really adamant at this point, to get SineReLU function to work so much, I will even write custom code in the keras-contrib library if that what takes to fix this error. (I could get away with Swish function but I won't)

## My Code:

Importing libraries

import tensorflow as tf
import numpy as np
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.layers import Conv2D, Dense, Flatten, MaxPool2D, Input, Dropout, CategoryEncoding, BatchNormalization, ReLU
from keras_contrib.layers.advanced_activations.sinerelu import SineReLU
from tensorflow.keras import Model
from keras.utils.generic_utils import get_custom_objects
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from keras.layers import Activation
from keras.layers.pooling.max_pooling2d import MaxPooling2D

# Calling custom functions
get_custom_objects().update({'SineReLU': Activation(SineReLU)})

Error Log:

    891         3, "Failed to convert %r to tensor: %s" % (type(value).__name__, e))
    892 
--> 893   raise TypeError(f"Could not build a TypeSpec for {value} of "
    894                   f"unsupported type {type(value)}.")
    895 

TypeError: Could not build a TypeSpec for <keras_contrib.layers.advanced_activations.sinerelu.SineReLU object at 0x7f4d5c575ad0> of unsupported type <class 'keras_contrib.layers.advanced_activations.sinerelu.SineReLU'>.

I will really appreciate if anyone can help me fix this error

- What I did I have added the method on how to add functions as activations and others from keras-contrib library using string alias

- How I did it I used get_custom_objects() function from keras.utils.generic_utils

- How you can verify it I added an example snippet code to teach how string alias method can be used to add functions from keras-contrib library in the README.md file. I don't think that requires to write a test as it only shows how to add string alias not creating or compiling an entire model.

This pull request fixes #issue_number_here

ImportError: cannot import name 'keras' from partially initialized module 'keras.api.v2' (most likely due to a circular import) (C:\Users\laptop media.conda\envs\tensorflow\lib\site-packages\keras\api_v2_init.py)

#when i write this code : "vgg16 = tf.keras.applications.VGG16(include_top=False) preprocess_input = tf.keras.applications.vgg16.preprocess_input image = tf.keras_preprocessing.image"

#ISSUE ImportError: cannot import name 'keras' from partially initialized module 'keras.api.v2' (most likely due to a circular import) (C:\Users\laptop media.conda\envs\tensorflow\lib\site-packages\keras\api_v2_init.py)

Hi， I am using CRF from keras_contrib,but I get this error " AttributeError: 'Tensor' object has no attribute '_keras_history'", when model.fit() is called. Any help would be highly appreciated This is my import import numpy as np import os import tensorflow as tf

from scipy.io import loadmat

from tensorflow import keras from tensorflow.keras import backend as K from tensorflow.keras.layers import Dense,Dropout,TimeDistributed,Bidirectional,LSTM,Input from tensorflow.keras.utils import to_categorical from tensorflow.keras.models import Sequential from tensorflow.keras.callbacks import ModelCheckpoint from keras_contrib.layers.crf import CRF from keras_contrib.losses import crf_loss from keras_contrib.metrics import crf_viterbi_accuracy from tensorflow.keras import Model This my partial code model = Sequential() model.add(Dense(4,activation='softmax')) model.add(CRF(4)) model.compile(optimizer='RMsprop', loss=crf_loss, metrics=[crf_viterbi_accuracy])

model.fit(train_data, train_label, batch_size=64, epochs=10,
                  validation_split=0.2)

model.summary()

Traceback (most recent call last): File "D:/CRF+LSTM/test1.py", line 426, in Linear_crf(train_data,train_label,test_data,test_label) File "D:/CRF+LSTM/test1.py", line 420, in Linear_crf validation_split=0.2, callbacks=[cp_callback]) File "D:\machine_learning\Anaconda_3\envs\cc\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1183, in fit tmp_logs = self.train_function(iterator) File "D:\machine_learning\Anaconda_3\envs\cc\lib\site-packages\tensorflow\python\eager\def_function.py", line 889, in call result = self._call(*args, **kwds) File "D:\machine_learning\Anaconda_3\envs\cc\lib\site-packages\tensorflow\python\eager\def_function.py", line 933, in _call self._initialize(args, kwds, add_initializers_to=initializers) File "D:\machine_learning\Anaconda_3\envs\cc\lib\site-packages\tensorflow\python\eager\def_function.py", line 764, in _initialize *args, **kwds)) File "D:\machine_learning\Anaconda_3\envs\cc\lib\site-packages\tensorflow\python\eager\function.py", line 3050, in _get_concrete_function_internal_garbage_collected graph_function, _ = self._maybe_define_function(args, kwargs) File "D:\machine_learning\Anaconda_3\envs\cc\lib\site-packages\tensorflow\python\eager\function.py", line 3444, in _maybe_define_function graph_function = self._create_graph_function(args, kwargs) File "D:\machine_learning\Anaconda_3\envs\cc\lib\site-packages\tensorflow\python\eager\function.py", line 3289, in _create_graph_function capture_by_value=self._capture_by_value), File "D:\machine_learning\Anaconda_3\envs\cc\lib\site-packages\tensorflow\python\framework\func_graph.py", line 999, in func_graph_from_py_func func_outputs = python_func(*func_args, **func_kwargs) File "D:\machine_learning\Anaconda_3\envs\cc\lib\site-packages\tensorflow\python\eager\def_function.py", line 672, in wrapped_fn out = weak_wrapped_fn().wrapped(*args, **kwds) File "D:\machine_learning\Anaconda_3\envs\cc\lib\site-packages\tensorflow\python\framework\func_graph.py", line 986, in wrapper raise e.ag_error_metadata.to_exception(e) AttributeError: in user code:

D:\machine_learning\Anaconda_3\envs\cc\lib\site-packages\tensorflow\python\keras\engine\training.py:855 train_function  *
    return step_function(self, iterator)
D:\machine_learning\Anaconda_3\envs\cc\lib\site-packages\keras_contrib-2.0.8-py3.7.egg\keras_contrib\losses\crf_losses.py:54 crf_loss  *
    crf, idx = y_pred._keras_history[:2]
D:\machine_learning\Anaconda_3\envs\cc\lib\site-packages\tensorflow\python\framework\ops.py:401 __getattr__
    self.__getattribute__(name)

AttributeError: 'Tensor' object has no attribute '_keras_history'

tensorflow-gpu==2.2.0 kears==2.3.1 keras_contrib=2.0.8 CUDA=10.1 cudnn=7.6.5 if I use from kears.layers import *,it not can use my GPU accelerate my code, use my CPU,but i use from tensorflow.keras.layers import * , it can use GPU accelerate, but have a error: The added layer must be an instance of class Layer. Found: <keras_contrib.layers.crf.CRF object at 0x12bfa4550>

Hello :) firstly thank you for this amazing work!

Could you explain (maybe with a source as well) why the normalization of the real data is defined in the range of [-1,1] here ?