for each block, for each layer, make 3 conv layers. These three conv layers constitute the concept of a block in the original paper. But you're repeating identical blocks.

So instead of there being 4 blocks, there are actually 12 in your example.

It seems your intention was to have 12 total layers, split into 4 blocks, but instead it seems you have 36 layers in 12 blocks.

This may be what you intended -- But the paper used 34 layers in most of it's experiments, so 36 is quite a jump ;).

Here's a tensorboard graph of one of the "blocks" so you can see what I'm referring to.

Hey, I'm getting an error in the convolutional autoencoder on the line tf.nn.deconv2d

"AttributeError: 'module' object has no attribute 'deconv2d'"

Also it isn't in the docs. https://www.tensorflow.org/versions/master/api_docs/python/nn.html

however there is tf.nn.conv2d_transpose https://www.tensorflow.org/versions/master/api_docs/python/nn.html#conv2d_transpose

Which is said to not be an actual deconvolution but just a transpose. Perhaps the method has recently been renamed to that?

Oddly, the 0.6 docs doesn't have conv2d_transpose or deconv2d https://www.tensorflow.org/versions/0.6.0/api_docs/python/nn.html

Perhaps they momentarily introduced deconv2d, then though it was inaccurate if it was only doing a transpose and not true deconv, and then renamed it?

I'm running HEAD as of a few days ago.

I have followed basic_convnet.py and when I run the code I got the error below:

C:\Users\Sami\Anaconda3\lib\site-packages\h5py_init_.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. from .conv import register_converters as register_converters Traceback (most recent call last): File "C:/Users/Sami/GitHub-Repository/my_project_github_pycharm/ML_algorithms_libs/ConvNet.py", line 2, in import tensorflow.examples.tutorials.mnist.input_data as input_data File "C:\Users\Sami\AppData\Roaming\Python\Python36\site-packages\tensorflow\examples\tutorials\mnist_init.py", line 21, in from tensorflow.examples.tutorials.mnist import input_data File "C:\Users\Sami\AppData\Roaming\Python\Python36\site-packages\tensorflow\examples\tutorials\mnist\input_data.py", line 29, in from tensorflow.contrib.learn.python.learn.datasets.mnist import read_data_sets File "C:\Users\Sami\AppData\Roaming\Python\Python36\site-packages\tensorflow\contrib_init.py", line 31, in from tensorflow.contrib import distributions File "C:\Users\Sami\AppData\Roaming\Python\Python36\site-packages\tensorflow\contrib\distributions_init_.py", line 33, in from tensorflow.contrib.distributions.python.ops.estimator import * File "C:\Users\Sami\AppData\Roaming\Python\Python36\site-packages\tensorflow\contrib\distributions\python\ops\estimator.py", line 21, in from tensorflow.contrib.learn.python.learn.estimators.head import compute_weighted_loss File "C:\Users\Sami\AppData\Roaming\Python\Python36\site-packages\tensorflow\contrib\learn_init.py", line 92, in from tensorflow.contrib.learn.python.learn import * File "C:\Users\Sami\AppData\Roaming\Python\Python36\site-packages\tensorflow\contrib\learn\python_init_.py", line 23, in from tensorflow.contrib.learn.python.learn import * File "C:\Users\Sami\AppData\Roaming\Python\Python36\site-packages\tensorflow\contrib\learn\python\learn_init_.py", line 25, in from tensorflow.contrib.learn.python.learn import estimators File "C:\Users\Sami\AppData\Roaming\Python\Python36\site-packages\tensorflow\contrib\learn\python\learn\estimators_init_.py", line 297, in from tensorflow.contrib.learn.python.learn.estimators.dnn import DNNClassifier File "C:\Users\Sami\AppData\Roaming\Python\Python36\site-packages\tensorflow\contrib\learn\python\learn\estimators\dnn.py", line 30, in from tensorflow.contrib.learn.python.learn.estimators import dnn_linear_combined File "C:\Users\Sami\AppData\Roaming\Python\Python36\site-packages\tensorflow\contrib\learn\python\learn\estimators\dnn_linear_combined.py", line 31, in from tensorflow.contrib.learn.python.learn.estimators import estimator File "C:\Users\Sami\AppData\Roaming\Python\Python36\site-packages\tensorflow\contrib\learn\python\learn\estimators\estimator.py", line 49, in from tensorflow.contrib.learn.python.learn.learn_io import data_feeder File "C:\Users\Sami\AppData\Roaming\Python\Python36\site-packages\tensorflow\contrib\learn\python\learn\learn_io_init_.py", line 21, in from tensorflow.contrib.learn.python.learn.learn_io.dask_io import extract_dask_data File "C:\Users\Sami\AppData\Roaming\Python\Python36\site-packages\tensorflow\contrib\learn\python\learn\learn_io\dask_io.py", line 26, in import dask.dataframe as dd File "C:\Users\Sami\Anaconda3\lib\site-packages\dask\dataframe_init_.py", line 3, in from .core import (DataFrame, Series, Index, _Frame, map_partitions, File "C:\Users\Sami\Anaconda3\lib\site-packages\dask\dataframe\core.py", line 41, in pd.core.computation.expressions.set_use_numexpr(False) AttributeError: module 'pandas.core.computation' has no attribute 'expressions'

Upgrading dask didn't help, I would like to know how to fix this issue.

Hi!

First of all, thanks for the tutorials, I could pick up tensorflow really fast because of them!

I was running the variational autoencoder example (11_variational_autoencoder.py) and noticed that the reported costs are "nan". I suspect the problem is in line 77, where "tanh" can create negative values for "y" that are later fed into "log" in lines 81 and 82. I suggest changing "tanh" in line 77 to "sigmoid" (I checked and it solves the problem).

Cheers,

Daniyar

Thanks for an awesome set of tutorials! I was tinkering with this a bit, trying to merge the chapter on convolutional autoencoders with tied weights (09) and the variational autoencoder (11). The adapted (messy!) code goes below. Everything works fine, except for the part where example reconstructions are made from latent representations: there, tensorflow now complains that I do not provide a value for x in the line: recon = sess.run(ae['y'], feed_dict={ae['z']: z}), which results in:

tensorflow.python.framework.errors.InvalidArgumentError: You must feed a value for placeholder tensor 'x' with dtype float
     [[Node: x = Placeholder[dtype=DT_FLOAT, shape=[], _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
etc.

So I started wondering: is it possible at all to use the architecture for this kind of reconstruction purpose (because of the weight tying?), or am I missing an obvious workaround?

Thanks in advance for any help!

import tensorflow as tf
import math
from libs.activations import lrelu
import numpy as np
from libs.utils import weight_variable, bias_variable, montage_batch


# %%
def VAE(input_shape=[None, 28, 28, 1],
        n_filters=[1, 10, 10, 10],
        filter_sizes=[3, 3, 3, 3],
        n_hidden=2,
        activation = lrelu):

    # %%
    # input to the network
    x = tf.placeholder(shape=input_shape, dtype=tf.float32, name='x')
    x_tensor = x
    current_input = x_tensor

    encoder, shapes = [], []
    for layer_i, n_output in enumerate(n_filters[1:]):
        n_input = current_input.get_shape().as_list()[3]
        shapes.append(current_input.get_shape().as_list())
        W = tf.Variable(
            tf.random_uniform([
                filter_sizes[layer_i],
                filter_sizes[layer_i],
                n_input, n_output],
                -1.0 / math.sqrt(n_input),
                1.0 / math.sqrt(n_input)))
        b = tf.Variable(tf.zeros([n_output]))
        encoder.append(W)
        output = lrelu(
            tf.add(tf.nn.conv2d(
                current_input, W, strides=[1, 2, 2, 1], padding='SAME'), b))
        current_input = output

    dims = current_input.get_shape().as_list()
    nb_flat = dims[1] * dims[2] * dims[3]

    flattened = tf.reshape(current_input, [-1, nb_flat])

    ###############################################
    W_mu = weight_variable([nb_flat, n_hidden])
    b_mu = bias_variable([n_hidden])

    W_log_sigma = weight_variable([nb_flat, n_hidden])
    b_log_sigma = bias_variable([n_hidden])

    z_mu = tf.matmul(flattened, W_mu) + b_mu
    z_log_sigma = 0.5 * (tf.matmul(flattened, W_log_sigma) + b_log_sigma)

    ###############################################

    ###############################################
    # %%
    # Sample from noise distribution p(eps) ~ N(0, 1)
    #epsilon = tf.random_normal(
    #        tf.pack([tf.shape(x)[0], n_hidden]))
    epsilon = tf.random_normal(
            [1, n_hidden])

    # Sample from posterior
    z = z_mu + tf.exp(z_log_sigma) * epsilon
    print(z.get_shape(), '+++')
    ###############################################

    W_ = tf.transpose(W_mu)
    b_ = tf.Variable(tf.zeros([nb_flat]))
    dense = tf.nn.tanh(tf.matmul(z, W_) + b_)

    current_input = tf.reshape(dense, [-1, dims[1], dims[2], dims[3]])
    print(dims)
    ###############################################

    encoder.reverse()
    shapes.reverse()

    # Build the decoder using the same weights
    for layer_i, shape in enumerate(shapes):
        W = encoder[layer_i]
        b = tf.Variable(tf.zeros([W.get_shape().as_list()[2]]))
        output = lrelu(tf.add(
            tf.nn.conv2d_transpose(
                current_input, W,
                tf.pack([tf.shape(x)[0], shape[1], shape[2], shape[3]]),
                strides=[1, 2, 2, 1], padding='SAME'), b))
        current_input = output

    # now have the reconstruction through the network
    y = current_input

    actual_cost = tf.reduce_sum(tf.square(y - x_tensor))
    kl_div = -0.5 * tf.reduce_sum(
        1.0 + 2.0 * z_log_sigma - tf.square(z_mu) - tf.exp(2.0 * z_log_sigma),
        1)

    loss = tf.reduce_mean(actual_cost + kl_div)

    ###############################################

    return {'cost': loss, 'x': x, 'z': z, 'y': y}


# %%
def test_mnist():
    """Summary

    Returns
    -------
    name : TYPE
        Description
    """
    # %%
    import tensorflow as tf
    import tensorflow.examples.tutorials.mnist.input_data as input_data
    import matplotlib.pyplot as plt

    # %%
    # load MNIST as before
    mnist = input_data.read_data_sets('MNIST_data', one_hot=True)
    ae = VAE()

    # %%
    learning_rate = 0.001
    optimizer = tf.train.AdamOptimizer(learning_rate).minimize(ae['cost'])

    # %%
    # We create a session to use the graph
    sess = tf.Session()
    sess.run(tf.initialize_all_variables())

    # %%
    # Fit all training data
    t_i = 0
    batch_size = 100
    n_epochs = 50
    n_examples = 10

    test_xs, _ = mnist.test.next_batch(n_examples)
    xs, ys = mnist.test.images, mnist.test.labels
    test_xs = test_xs.reshape((n_examples, 28, 28, 1))
    xs = xs.reshape((xs.shape[0], 28, 28, 1))

    fig_manifold, ax_manifold = plt.subplots(1, 1)
    fig_reconstruction, axs_reconstruction = plt.subplots(2, n_examples, figsize=(10, 2))
    fig_image_manifold, ax_image_manifold = plt.subplots(1, 1)

    for epoch_i in range(n_epochs):

        print('--- Epoch', epoch_i)

        train_cost = 0

        for batch_i in range(mnist.train.num_examples // batch_size):
            batch_xs, _ = mnist.train.next_batch(batch_size)
            batch_xs = batch_xs.reshape((batch_size, 28, 28, 1))
            train_cost += sess.run([ae['cost'], optimizer],
                                   feed_dict={ae['x']: batch_xs,})[0]

            if batch_i % 20 == 0:
                # Plot example reconstructions from latent layer
                imgs = []
                for img_i in np.linspace(-3, 3, n_examples):
                    for img_j in np.linspace(-3, 3, n_examples):
                        z = np.array([[img_i, img_j]], dtype=np.float32)
                        print(z)
                        print(z.dtype)
                        recon = sess.run(ae['y'], feed_dict={ae['z']: z})
                        imgs.append(np.reshape(recon, (1, 28, 28, 1)))

                imgs_cat = np.concatenate(imgs)
                ax_manifold.imshow(montage_batch(imgs_cat))
                fig_manifold.savefig('vizes/manifold_%08d.png' % t_i)

                # Plot example reconstructions
                recon = sess.run(ae['y'], feed_dict={ae['x']: test_xs})

                for example_i in range(n_examples):
                    axs_reconstruction[0][example_i].imshow(
                        np.reshape(test_xs[example_i, :], (28, 28)))
                    axs_reconstruction[1][example_i].imshow(
                        np.reshape(
                            np.reshape(recon[example_i, ...], (784,)),
                            (28, 28)))
                    axs_reconstruction[0][example_i].axis('off')
                    axs_reconstruction[1][example_i].axis('off')
                fig_reconstruction.savefig('vizes/reconstruction_%08d.png' % t_i)

                # %%
                # Plot manifold of latent layer
                zs = sess.run(ae['z'], feed_dict={ae['x']: xs})
                ax_image_manifold.clear()
                ax_image_manifold.scatter(zs[:, 0], zs[:, 1],
                    c=np.argmax(ys, 1), alpha=0.2)
                ax_image_manifold.set_xlim([-6, 6])
                ax_image_manifold.set_ylim([-6, 6])
                ax_image_manifold.axis('off')
                fig_image_manifold.savefig('vizes/image_manifold_%08d.png' % t_i)

                t_i += 1


        print('Train cost:', train_cost /
              (mnist.train.num_examples // batch_size))

        valid_cost = 0
        for batch_i in range(mnist.validation.num_examples // batch_size):
            batch_xs, _ = mnist.validation.next_batch(batch_size)
            batch_xs = batch_xs.reshape((batch_size, 28, 28, 1))
            valid_cost += sess.run([ae['cost']],
                                   feed_dict={ae['x']: batch_xs})[0]
        print('Validation cost:', valid_cost /
              (mnist.validation.num_examples // batch_size))


if __name__ == '__main__':
    test_mnist()

I read the code in batch_norm, and found that self.ema (the ExponentialMovingAverage object) is never used. I thought at some point we need to call self.ema.apply([self.mean, slef.variance]) to create the updater of the moving average? The moving average will be used in validation/testing stage, instead of the batch mean/variance, I think.

Hi! I was very interested in your tutorials about tensorflow and i decided to translate it on russian language. Now i have translated only 01_basic.py, also i want to translate other files.

May i push it in your or my repository?

1. This jupyter notebook now works with the most recent versions of tensorflow.
2. The final picture is now produced using both the eval() as well as the session.run() method.

Thank you for the tutorials!

Since Tensorflow 1.0.0, tf.neg is deprecated in favor of tf.negative, see https://www.tensorflow.org/api_docs/python/tf/negative and https://github.com/tensorflow/tensorflow/releases

Hi @pkmital Thanks a lot for the great collection. Regarding to convolutional autoencoder example, I was wondering how it would be possible to add maxpooling and maxunpoling layers? The problem is that I am using your code for relatively large images (e.g. 250x250) and am going to use the latent representation layer for visualization by something like T-SNE.
Further, based on this work seems having maxpooling is essential in order to have plausible filters.

I found discussions and some solutions here and here, but could not yet figure it out how to extend your code. Thanks a lot for any advice.

The documentation on utils.montage and utils.montage_batch suggests that these functions work with tensors as input. However, they actually receive a numpy array. For handling tensors as inputs and outputs (e.g. when creating TF summaries), it would have to be wrapped around tf.py_func. Here's a PR to fix the documentation.

Thank you for this tutorial, it was a helpful resource.

Hi, thanks for sharing these,

I noticed in the convolutional autoencoder that the default number of filters for the first layer is 1:

https://github.com/pkmital/tensorflow_tutorials/blob/master/python/09_convolutional_autoencoder.py#L14

Why is that? Wouldn't that mean we aren't interested in identifying features in the first layer and just letting the model find a suitable "brush" that works everywhere in that layer?

Thanks again!

hello~ when I'm running 06-modern-convnet with tensorflow 1.4, it presents an error. Could you please tell me how to solve it? My friends also have this prolbem. Traceback (most recent call last): File "C:/zwPython/py_demo/tf_demo/06_modern_convnet.py", line 35, in is_training, scope='bn1'), name='lrelu1') File "C:\zwPython\py_demo\tf_demo\libs\batch_norm.py", line 58, in batch_norm lambda: (ema_mean, ema_var)) File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\util\deprecation.py", line 316, in new_func return func(*args, **kwargs) File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\ops\control_flow_ops.py", line 1864, in cond orig_res_f, res_f = context_f.BuildCondBranch(false_fn) File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\ops\control_flow_ops.py", line 1729, in BuildCondBranch result = nest.map_structure(self._BuildCondTensor, original_result) File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\util\nest.py", line 413, in map_structure structure[0], [func(*x) for x in entries]) File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\util\nest.py", line 413, in structure[0], [func(*x) for x in entries]) File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\ops\control_flow_ops.py", line 1721, in _BuildCondTensor return self._ProcessOutputTensor(ops.convert_to_tensor(v)) File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\framework\ops.py", line 836, in convert_to_tensor as_ref=False) File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\framework\ops.py", line 926, in internal_convert_to_tensor ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref) File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\framework\constant_op.py", line 229, in _constant_tensor_conversion_function return constant(v, dtype=dtype, name=name) File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\framework\constant_op.py", line 208, in constant value, dtype=dtype, shape=shape, verify_shape=verify_shape)) File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\framework\tensor_util.py", line 371, in make_tensor_proto raise ValueError("None values not supported.") ValueError: None values not supported.

Hi, I just wanted to ensure that the mean subtraction in the Convolution Autoencoder's test images are correct. Here the mean of the train image is being subtracted from the test images. Correct me if I am wrong, else i'd open a pull request to fix this :)

Cheers, Ramana

I feel that ~~tf.get_variable()~~ has to be used instead of ~~tf.Variable()~~. In each epoch all the mini batches have to interact with same set of weights. But since we are using ~~tf.Variable()~~ for each ~~autoencoder['cost']~~ calculation, we indirectly call the autoencoder function which creates a new set of weights every time we call pkmital@

I am getting TypeError: pred must not be a Python bool

in the line lambda: (ema_mean, ema_var)) in batch_norm function.

Looking forward for the help

Above error is while using tf.nn.conv2d_transpose and then batch norm