A simple Tensorflow based library for deep and/or denoising AutoEncoder.

Rajarshee Mitra

Last update: Nov 18, 2022

Overview

libsdae - deep-Autoencoder & denoising autoencoder

A simple Tensorflow based library for Deep autoencoder and denoising AE. Library follows sklearn style.

Prerequisities & Support

Tensorflow 1.0 is needed.
Supports both Python 2.7 and 3.4+ . Inform if it doesn't.

Installing

pip install git+https://github.com/rajarsheem/libsdae.git

Usage and small doc

test.ipynb has small example where both a tiny and a large dataset is used.

from deepautoencoder import StackedAutoEncoder
model = StackedAutoEncoder(dims=[5,6], activations=['relu', 'relu'], noise='gaussian', epoch=[10000,500],
                            loss='rmse', lr=0.007, batch_size=50, print_step=2000)
# usage 1 - encoding same data                           
result = model.fit_transform(x)
# usage 2 - fitting on one dataset and transforming (encoding) on another data
model.fit(x)
result = model.transform(np.random.rand(5, x.shape[1]))

Important points:

If noise is not given, it becomes an autoencoder instead of denoising autoencoder.
dims refers to the dimenstions of hidden layers. (3 layers in this case)
noise = (optional)['gaussian', 'mask-0.4']. mask-0.4 means 40% of bits will be masked for each example.
x_ is the encoded feature representation of x.
loss = (optional) reconstruction error. rmse or softmax with cross entropy are allowed. default is rmse.
print_step is the no. of steps to skip between two loss prints.
activations can be 'sigmoid', 'softmax', 'tanh' and 'relu'.
batch_size is the size of batch in every epoch
Note that while running, global loss means the loss on the total dataset and not on a specific batch.
epoch is a list denoting the no. of iterations for each layer.

Citing

Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion by P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio and P. Manzagol (Journal of Machine Learning Research 11 (2010) 3371-3408)

Contributing

You are free to contribute by starting a pull request. Some suggestions are:

Variational Autoencoders
Recurrent Autoencoders.

Comments

Cannot get cross-entropy to work
Hi,

I'm playing around with this code for a research project and everything works fine with mean-squared-error, however as soon as I switch to cross-entropy (which I really want), it does not converge and loss gets bigger over time... I tried numerous parameters but nothing seems to work. I'm using MNIST with the following model.

model = StackedAutoEncoder( dims=[100], activations=['softmax'], noise='gaussian', epoch=[1000], loss='cross-entropy', lr=0.005, batch_size=100, print_step=100 )

Do you know why this isn't working?

Thanks!
opened by elggem 14
Error in Mini Batch
The mini batch function may generate same sample over There is no removal already used indices

Also issue with mask noise

Following error:

File "algorithms/deepautoencoder/stacked_autoencoder.py", line 42, in add_noise frac = float(self.noise.split('-')[1])
opened by nthakor 4
Blas GEMM launch failed

Hi, Rajarshee Mitra! Thank you very much for your code! But when I tried running your code on my computer, I always got the error shown below:

''' InternalError: Blas GEMM launch failed : a.shape=(100, 784), b.shape=(784, 200), m=100, n=200, k=784 [[Node: MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/gpu:0"](_arg_x_0_0/_7, Variable/read)]] ''' where 100 is batch_size, 784 and 200 are dimensions.

Can you offer me some help! Thanks a lot! Phil

opened by kxxwz 3
AutoEncoder Graph Construction

Hi Rajarshee Mitra,

I have question regarding your code. I am confused how your code construct the connections between layers. Specifically, in line 63 of stacked_autoencoder.py, you called the run function for different layers . it seems for me that you construct n different auto-encoders instead of one stacked with n hidden layers.

Best regards, Mohammad

opened by akbari59 3
Other dataset than mnist

I have just started learning denoising autoencoders but I want to import dataset of images than mnist what can be done to do so . I have tried giving path name and assigning it to a variable but im getting a type error.

opened by PatelAxita 2
show validation loss during training

Hey, I enhanced the code, so it can show the validation loss additional to the training loss. This is optional and will be only displayed, if a test dataset is given to the fit() or fit_transform() method as argument (beside the training dataset). I think the validation loss is another important metric to evaluate the model. Please tell me if there are some enhancements to my code.

Regards Feserich

opened by Feserich 2
Evaluation
Nice library. How did you evaluate the goodness of your feature representation? Is the global loss the general reconstruction? So for example, the 2nd DAE gets the encoded version of the image, so is the loss function of the 2nd DAE the followin: LOSS(decoded(noisy encoded input), decoded(encoded output)) This way the network doesn't loose seight of the global goal: Finding an optimal feature representation, right?

What I did was:

Train first DAE, get encoded versions of the training data

Use the encoded version to train another DAE with LOSS(noisy encoded, encoded) I think it was my mistake that I don't consider the completely reconstructed image as the loss..

Is that right?
opened by jubueche 1

A simple Tensorflow based library for deep and/or denoising AutoEncoder.

Related tags

Overview

libsdae - deep-Autoencoder & denoising autoencoder

Prerequisities & Support

Installing

Usage and small doc

Important points:

Citing

Contributing

Comments

Cannot get cross-entropy to work

Error in Mini Batch

Blas GEMM launch failed

AutoEncoder Graph Construction

Other dataset than mnist

show validation loss during training

Evaluation

Owner

Rajarshee Mitra

Official Implementation of Swapping Autoencoder for Deep Image Manipulation (NeurIPS 2020)

sequitur is a library that lets you create and train an autoencoder for sequential data in just two lines of code

Code image classification of MNIST dataset using different architectures: simple linear NN, autoencoder, and highway network

Official implementation of Deep Convolutional Dictionary Learning for Image Denoising.

Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising

Minimal diffusion models - Minimal code and simple experiments to play with Denoising Diffusion Probabilistic Models (DDPMs)

Speech Enhancement Generative Adversarial Network Based on Asymmetric AutoEncoder

EDCNN: Edge enhancement-based Densely Connected Network with Compound Loss for Low-Dose CT Denoising

Video Autoencoder: self-supervised disentanglement of 3D structure and motion

Project looking into use of autoencoder for semi-supervised learning and comparing data requirements compared to supervised learning.

Code of 3D Shape Variational Autoencoder Latent Disentanglement via Mini-Batch Feature Swapping for Bodies and Faces

Clockwork Variational Autoencoder

VITS: Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech

Implementation for "Manga Filling Style Conversion with Screentone Variational Autoencoder" (SIGGRAPH ASIA 2020 issue)

MADE (Masked Autoencoder Density Estimation) implementation in PyTorch

Molecular AutoEncoder in PyTorch

Recurrent Variational Autoencoder that generates sequential data implemented with pytorch