A PyTorch Implementation of "Neural Arithmetic Logic Units"

Last update: Nov 18, 2022

Overview

Neural Arithmetic Logic Units

[WIP]

This is a PyTorch implementation of Neural Arithmetic Logic Units by Andrew Trask, Felix Hill, Scott Reed, Jack Rae, Chris Dyer and Phil Blunsom.

API

from models import *

# single layer modules
NeuralAccumulatorCell(in_dim, out_dim)
NeuralArithmeticLogicUnitCell(in_dim, out_dim)

# stacked layers
NAC(num_layers, in_dim, hidden_dim, out_dim)
NALU(num_layers, in_dim, hidden_dim, out_dim)

Experiments

To reproduce "Numerical Extrapolation Failures in Neural Networks" (Section 1.1), run:

python failures.py

This should generate the following plot:

To reproduce "Simple Function Learning Tasks" (Section 4.1), run:

python function_learning.py

This should generate a text file called interpolation.txt with the following results. (Currently only supports interpolation, I'm working on the rest)

	Relu6	None	NAC	NALU
a + b	4.472	0.132	0.154	0.157
a - b	85.727	2.224	2.403	34.610
a * b	89.257	4.573	5.382	1.236
a / b	97.070	60.594	5.730	3.042
a ^ 2	89.987	2.977	4.718	1.117
sqrt(a)	5.939	40.243	7.263	1.119

Notes

RMSprop works really well with NAC and NALU
high learning rate (0.01) does a good job as well

A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch

This repository holds NVIDIA-maintained utilities to streamline mixed precision and distributed training in Pytorch. Some of the code here will be included in upstream Pytorch eventually. The intention of Apex is to make up-to-date utilities available to users as quickly as possible.

6.9k Jan 3, 2023

Objective of the repository is to learn and build machine learning models using Pytorch. 30DaysofML Using Pytorch

PyTorch-LIT is the Lite Inference Toolkit (LIT) for PyTorch which focuses on easy and fast inference of large models on end-devices.

PyTorch-LIT PyTorch-LIT is the Lite Inference Toolkit (LIT) for PyTorch which focuses on easy and fast inference of large models on end-devices. With

157 Dec 11, 2022

Comments

Calculating W

https://github.com/kevinzakka/NALU-pytorch/blob/002c97f21ebd067b577e9c218b154cf68e867d31/models/nalu.py#L28

Isn't it that according to the paper, W is calculated as below ?: W = tanh(Wˆ ) * σ(Mˆ )

And shouldn't the NALU's "a" NAC and the "m" NAC share the W ?: I quote from the paper:

(NALU) uses two NACs with tied weights ...

opened by pushkarparanjpe 2
NAC fix

The NeuralAccumulatorCell class has a bug in it that turns it into a simple linear cell. The way it's written right now, the W_hat and M_hat are not updated and the tanh and sigmoid don't do anything.

I go into more detail in this notebook: https://github.com/vrxacs/NALU/blob/master/fauxNAC.ipynb

opened by vrxacs 1
Clarification on Multiplication results

According to the functional learning results, NAC performs comparable to NALU on the multiplication task. However, there's no means that NAC can learn multiplication. Indeed in the original paper's results, NAC performs even worse than ReLU6. Do you have idea what's happening here?

opened by ssydasheng 0
The generation of training data is wrong

According to the paper, for the simple function learning task, we first generate n, m, p, q randomly. But they are consistent for all training samples. (Appendix B)

But in your code, you use different n, m, p, q for every record.

https://github.com/kevinzakka/NALU-pytorch/blob/8ddca091f1282575766e9aaaa7657a19385873ef/function_learning.py#L31-L32

This doesn't make sense to me.

I think the design of training data is to let the first NALU layer learn how to add input and get a, b, and let the second NALU layer perform the arithmetic operator (i.e. a+b, a-b). So n, m, p, q should be consistent for all training samples. Otherwise, NALU learns nothing.

opened by merrymercy 1

A PyTorch Implementation of "Neural Arithmetic Logic Units"

Related tags

Overview

Neural Arithmetic Logic Units

API

Experiments

Notes

You might also like...

A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch

Objective of the repository is to learn and build machine learning models using Pytorch. 30DaysofML Using Pytorch

Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks

The Incredible PyTorch: a curated list of tutorials, papers, projects, communities and more relating to PyTorch.

Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks

A bunch of random PyTorch models using PyTorch's C++ frontend

PyTorch Autoencoders - Implementing a Variational Autoencoder (VAE) Series in Pytorch.

PyTorch-LIT is the Lite Inference Toolkit (LIT) for PyTorch which focuses on easy and fast inference of large models on end-devices.

Comments

Calculating W

NAC fix

Clarification on Multiplication results

The generation of training data is wrong

Owner

Kevin Zakka

An essential implementation of BYOL in PyTorch + PyTorch Lightning

RealFormer-Pytorch Implementation of RealFormer using pytorch

A PyTorch implementation of the paper Mixup: Beyond Empirical Risk Minimization in PyTorch

A pytorch implementation of Pytorch-Sketch-RNN

PyTorch implementation of Advantage async actor-critic Algorithms (A3C) in PyTorch

Pytorch-diffusion - A basic PyTorch implementation of 'Denoising Diffusion Probabilistic Models'

RetinaNet-PyTorch - A RetinaNet Pytorch Implementation on remote sensing images and has the similar mAP result with RetinaNet in MMdetection

RETRO-pytorch - Implementation of RETRO, Deepmind's Retrieval based Attention net, in Pytorch

HashNeRF-pytorch - Pure PyTorch Implementation of NVIDIA paper on Instant Training of Neural Graphics primitives

Generic template to bootstrap your PyTorch project with PyTorch Lightning, Hydra, W&B, and DVC.