Dilated RNNs in pytorch

Related tags

Deep Learning machine-learning deep-learning pytorch rnns
Overview

PyTorch Dilated Recurrent Neural Networks

PyTorch implementation of Dilated Recurrent Neural Networks (DilatedRNN).

Getting Started

Installation:

$ pip3 install -r requirements.txt

Run the tests:

$ python tests.py

Example

Define a dilated RNN based on GRU cells with 9 layers, dilations 1, 2, 4, 8, 16, ... Then pass the hidden state to a further update

import drnn
import torch

n_input = 20
n_hidden = 32
n_layers = 9
cell_type = 'GRU'

model = drnn.DRNN(n_input, n_hidden, n_layers, cell_type)

x1 = torch.randn(23, 2, n_input)
x2 = torch.randn(23, 2, n_input)

out, hidden = model(x1)
out, hidden = model(x2, hidden)

Copy Task

$ python3 -m copy_memory.copymem_test --help

Character LeveL RNN

$ python3 -m char_rnn.char_rnn_test --help
Comments
  • Inefficient usage of resources

    Inefficient usage of resources

    I was exploring this project for a general purpose forecasting model project I was working on, and I realized that the performance of this model with multiple layers (say 5) is actually worse in both forward and backward passes than a standard pytorch GRU module with 5 layers.

    https://gist.github.com/zeryx/c43fc53b4d3f71c4942dff44912aa3cb

    From my understanding of the paper, the DRNN module should be at least as performant as the equivalent GRU module if not dramatically superior in both forward and backward pass compute time.

    opened by zeryx 5
  • Support CPU-only QRNN.

    Support CPU-only QRNN.

    The previous build previously propagate the use or disuse of CPU-only QRNNs to model creation.

    This has been corrected, and I've added a unit test to ensure that it runs as expected.

    opened by dnbaker 1
  • Get hidden memory to work properly

    Get hidden memory to work properly

    as I described in https://github.com/zalandoresearch/pt-dilate-rnn/issues/12, the hidden memory tensor was not being updated properly. Some of the changes I made were personal QoL improvements such as renaming, not reusing variables for different purposes, etc. Can edit if desired.

    opened by zeryx 1
  • Need to mention Dropout

    Need to mention Dropout

    This should be the line model = drnn.DRNN(n_input, n_hidden, n_layers, dropout, cell_type) or model = drnn.DRNN(n_input=n_input, n_hidden=n_hidden, n_layers=n_layers, cell_type=cell_type)

    otherwise following error: ValueError: could not convert string to float: 'GRU'

    opened by yuviabhi 0
  • Question about Parameter-size

    Question about Parameter-size

    Hello! Thank you very much for sharing this work. In Table 2 of the paper, GRU and 5-layer Dilated GRU has the same number of parameter-size. Can you please help me to understand how this is possible? Would they also have the same GPU memory requirement, despite significantly higher number of layers of the Dilated version?

    I would also like to take some advise on whether Dilated GRU also be helpful when the sequence-length is not too high e.g. smaller than 32? Is there any way to pick the maximum number of layers to experiment with based on the sequence-length? Lastly, how can I most easily modify your code so that it becomes stateless? Thanks again!

    opened by kayuksel 0
Owner
Zalando Research
Repositories of the research branch of Zalando SE
Zalando Research
GitHub
PyTorch version repo for CSRNet: Dilated Convolutional Neural Networks for Understanding the Highly Congested Scenes

Study-CSRNet-pytorch This is the PyTorch version repo for CSRNet: Dilated Convolutional Neural Networks for Understanding the Highly Congested Scenes

null 0 Mar 1, 2022
Classify bird species based on their songs using SIamese Networks and 1D dilated convolutions.

The goal is to classify different birds species based on their songs/calls. Spectrograms have been extracted from the audio samples and used as features for classification.

Aditya Dutt 9 Dec 27, 2022
Official code for "Stereo Waterdrop Removal with Row-wise Dilated Attention (IROS2021)"

Stereo-Waterdrop-Removal-with-Row-wise-Dilated-Attention This repository includes official codes for "Stereo Waterdrop Removal with Row-wise Dilated A

null 29 Oct 1, 2022
An essential implementation of BYOL in PyTorch + PyTorch Lightning

Essential BYOL A simple and complete implementation of Bootstrap your own latent: A new approach to self-supervised Learning in PyTorch + PyTorch Ligh

Enrico Fini 48 Sep 27, 2022
RealFormer-Pytorch Implementation of RealFormer using pytorch

RealFormer-Pytorch Implementation of RealFormer using pytorch. Includes comparison with classical Transformer on image classification task (ViT) wrt C

Simo Ryu 90 Dec 8, 2022
Generic template to bootstrap your PyTorch project with PyTorch Lightning, Hydra, W&B, and DVC.

NN Template Generic template to bootstrap your PyTorch project. Click on Use this Template and avoid writing boilerplate code for: PyTorch Lightning,

Luca Moschella 520 Dec 30, 2022
A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch

This repository holds NVIDIA-maintained utilities to streamline mixed precision and distributed training in Pytorch. Some of the code here will be included in upstream Pytorch eventually. The intention of Apex is to make up-to-date utilities available to users as quickly as possible.

NVIDIA Corporation 6.9k Jan 3, 2023
Objective of the repository is to learn and build machine learning models using Pytorch. 30DaysofML Using Pytorch

30 Days Of Machine Learning Using Pytorch Objective of the repository is to learn and build machine learning models using Pytorch. List of Algorithms

Mayur 119 Nov 24, 2022
Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

Pytorch Lightning 1.4k Jan 1, 2023
Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks

Amazon Forest Computer Vision Satellite Image tagging code using PyTorch / Keras Here is a sample of images we had to work with Source: https://www.ka

Mamy Ratsimbazafy 360 Dec 10, 2022
The Incredible PyTorch: a curated list of tutorials, papers, projects, communities and more relating to PyTorch.

This is a curated list of tutorials, projects, libraries, videos, papers, books and anything related to the incredible PyTorch. Feel free to make a pu

Ritchie Ng 9.2k Jan 2, 2023
A PyTorch implementation of the paper Mixup: Beyond Empirical Risk Minimization in PyTorch

Mixup: Beyond Empirical Risk Minimization in PyTorch This is an unofficial PyTorch implementation of mixup: Beyond Empirical Risk Minimization. The co

Harry Yang 121 Dec 17, 2022
A pytorch implementation of Pytorch-Sketch-RNN

Pytorch-Sketch-RNN A pytorch implementation of https://arxiv.org/abs/1704.03477 In order to draw other things than cats, you will find more drawing da

Alexis David Jacq 172 Dec 12, 2022
PyTorch implementation of Advantage async actor-critic Algorithms (A3C) in PyTorch

Advantage async actor-critic Algorithms (A3C) in PyTorch @inproceedings{mnih2016asynchronous, title={Asynchronous methods for deep reinforcement lea

LEI TAI 111 Dec 8, 2022
Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks

Amazon Forest Computer Vision Satellite Image tagging code using PyTorch / Keras Here is a sample of images we had to work with Source: https://www.ka

Mamy Ratsimbazafy 359 Jan 5, 2023
A bunch of random PyTorch models using PyTorch's C++ frontend

PyTorch Deep Learning Models using the C++ frontend Gettting started Clone the repo 1. https://github.com/mrdvince/pytorchcpp 2. cd fashionmnist or

Vince 0 Jul 13, 2021
PyTorch Autoencoders - Implementing a Variational Autoencoder (VAE) Series in Pytorch.

PyTorch Autoencoders Implementing a Variational Autoencoder (VAE) Series in Pytorch. Inspired by this repository Model List check model paper conferen

Subin An 8 Nov 21, 2022
PyTorch-LIT is the Lite Inference Toolkit (LIT) for PyTorch which focuses on easy and fast inference of large models on end-devices.

PyTorch-LIT PyTorch-LIT is the Lite Inference Toolkit (LIT) for PyTorch which focuses on easy and fast inference of large models on end-devices. With

Amin Rezaei 157 Dec 11, 2022
A general framework for deep learning experiments under PyTorch based on pytorch-lightning

torchx Torchx is a general framework for deep learning experiments under PyTorch based on pytorch-lightning. TODO list gan-like training wrapper text

Yingtian Liu 6 Mar 17, 2022