The dynamics of representation learning in shallow, non-linear autoencoders

Maria Refinetti

Last update: Jun 8, 2022

Related tags

Deep Learning NonLinearShallowAE

Overview

The dynamics of representation learning in shallow, non-linear autoencoders

The package is written in python and uses the pytorch implementation to ML. Repository src contains the source files.

train_autoencoder.py
Trains (online and off-line) a shallow autoencoder both on a synthetic dataset and on benchmark datasets. Includes an implementation of the analytical equations allowing to track the dynamics of online training at all times. Included benchmark datasets : Cifar 10 (in gray scale) and FashionMNIST

Example of command to train online, integrating the analytical equations, on a synthetic dataset:
python3 train_autoencoder.py --D 500 --K 2 --dataset sinusoidal --analytical_updates 1

Example of command to train on finite dataset:
python3 train_autoencoder.py --dataset fmnist

truncated_vanilla_SGD.py
Trains an AE using different learning rules for reconstruction:

sanger's rule
vanilla SGD
truncated version of SGD introduced in the article Includes the implementation of the analytical equations tracking the dynamics of learning.

Example of command to train online, integrating the analytical equations, on a synthetic dataset:
python3 truncated_vanilla_SGD.py --K 5 --low_rank 5 --D 500 --analytical 1

pytorch implementation of "Contrastive Multiview Coding", "Momentum Contrast for Unsupervised Visual Representation Learning", and "Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination"

Unofficial implementation: MoCo: Momentum Contrast for Unsupervised Visual Representation Learning (Paper) InsDis: Unsupervised Feature Learning via N

16 Nov 4, 2020

A Fast Monotone Rotating Shallow Water model

pyRSW A Fast Monotone Rotating Shallow Water model How fast? As fast as a sustained 2 Gflop/s per core on a 2.5 GHz cpu (or 2048 Gflop/s with 1024 cor

13 Sep 28, 2022

Shallow Convolutional Neural Networks for Human Activity Recognition using Wearable Sensors

-IEEE-TIM-2021-1-Shallow-CNN-for-HAR [IEEE TIM 2021-1] Shallow Convolutional Neural Networks for Human Activity Recognition using Wearable Sensors All

1 May 17, 2022

Event sourced bank - A wide-and-shallow example using the Python event sourcing library

Event Sourced Bank A "wide but shallow" example of using the Python event sourci

3 Mar 9, 2022

A Robust Non-IoU Alternative to Non-Maxima Suppression in Object Detection

Confluence: A Robust Non-IoU Alternative to Non-Maxima Suppression in Object Detection 1. 介绍用以替代 NMS，在所有 bbox 中挑选出最优的集合。 NMS 仅考虑了 bbox 的得分，然后根据 IOU 来

44 Sep 15, 2022

PyBullet CartPole and Quadrotor environments—with CasADi symbolic a priori dynamics—for learning-based control and reinforcement learning

safe-control-gym Physics-based CartPole and Quadrotor Gym environments (using PyBullet) with symbolic a priori dynamics (using CasADi) for learning-ba

300 Dec 28, 2022

Code for "Contextual Non-Local Alignment over Full-Scale Representation for Text-Based Person Search"

Contextual Non-Local Alignment over Full-Scale Representation for Text-Based Person Search This is an implementation for our paper Contextual Non-Loca

50 Dec 3, 2022

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

H2O H2O is an in-memory platform for distributed, scalable machine learning. H2O uses familiar interfaces like R, Python, Scala, Java, JSON and the Fl

6.1k Jan 5, 2023

Machine Learning From Scratch. Bare bones NumPy implementations of machine learning models and algorithms with a focus on accessibility. Aims to cover everything from linear regression to deep learning.

Machine Learning From Scratch About Python implementations of some of the fundamental Machine Learning models and algorithms from scratch. The purpose

21.8k Jan 9, 2023

The dynamics of representation learning in shallow, non-linear autoencoders

Related tags

Overview

The dynamics of representation learning in shallow, non-linear autoencoders

You might also like...

pytorch implementation of "Contrastive Multiview Coding", "Momentum Contrast for Unsupervised Visual Representation Learning", and "Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination"

A Fast Monotone Rotating Shallow Water model

Shallow Convolutional Neural Networks for Human Activity Recognition using Wearable Sensors

Event sourced bank - A wide-and-shallow example using the Python event sourcing library

A Robust Non-IoU Alternative to Non-Maxima Suppression in Object Detection

PyBullet CartPole and Quadrotor environments—with CasADi symbolic a priori dynamics—for learning-based control and reinforcement learning

Code for "Contextual Non-Local Alignment over Full-Scale Representation for Text-Based Person Search"

Machine Learning From Scratch. Bare bones NumPy implementations of machine learning models and algorithms with a focus on accessibility. Aims to cover everything from linear regression to deep learning.

Owner

Maria Refinetti

We have implemented shaDow-GNN as a general and powerful pipeline for graph representation learning. For more details, please find our paper titled Deep Graph Neural Networks with Shallow Subgraph Samplers, available on arXiv (https//arxiv.org/abs/2012.01380).

A framework that constructs deep neural networks, autoencoders, logistic regressors, and linear networks

PyTorch Code of "Memory In Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity from Spatiotemporal Dynamics"

With this package, you can generate mixed-integer linear programming (MIP) models of trained artificial neural networks (ANNs) using the rectified linear unit (ReLU) activation function

Simple Linear 2nd ODE Solver GUI - A 2nd constant coefficient linear ODE solver with simple GUI using euler's method

Hitters Linear Regression - Hitters Linear Regression With Python

Pytorch implementation of CVPR2020 paper “VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation”

PyTorch implementation of the WarpedGANSpace: Finding non-linear RBF paths in GAN latent space (ICCV 2021)

The official repo for OC-SORT: Observation-Centric SORT on video Multi-Object Tracking. OC-SORT is simple, online and robust to occlusion/non-linear motion.