Package for working with hypernetworks in PyTorch.

Related tags

Deep Learning hypnettorch
Overview

Hypernetworks for PyTorch

This package contains utilities that make it easy to work with hypernetworks in PyTorch.

Installation

You can either install the latest package version via

python3 -m pip install hypnettorch

or, you directly install the current sources

python3 -m pip install git+https://github.com/chrhenning/hypnettorch

Installation for developers

If you actively develop the package, it is easiest to install it in development mode, such that all changes that are done to source files are directly visible when you use the package.

Clone the repository to a location of your choice

git clone https://github.com/chrhenning/hypnettorch.git

and move inside the cloned repo

cd ./hypnettorch

Now, you can simply install the package in editable mode, which will ensure that you can easily update the package sources (cf. development mode)

pip3 install --editable . --user

Since the package was installed in editable mode, you can always update the sources simply by pulling the most recent code

git pull

You can uninstall the package at any point by running python3 setup.py develop -u.

Usage

The basic functionalities of the package are quite intuitive and easy to use, e.g.,

from hypnettorch.mnets import MLP
net = MLP()

There are several tutorials. Check out the getting started tutorial when working with hypnettorch for the first time.

Documentation

The documentation can be found here.

Note for developers

The documentation can be build using

python3 setup.py build_sphinx

and opened via the file index.html.

Citation

When using this package in your research project, please consider citing one of our papers for which this package has been developed.

@inproceedings{oshg2019hypercl,
title={Continual learning with hypernetworks},
author={Johannes von Oswald and Christian Henning and Jo{\~a}o Sacramento and Benjamin F. Grewe},
booktitle={International Conference on Learning Representations},
year={2020},
url={https://arxiv.org/abs/1906.00695}
}

@inproceedings{ehret2020recurrenthypercl,
  title={Continual Learning in Recurrent Neural Networks},
  author={Benjamin Ehret and Christian Henning and Maria R. Cervera and Alexander Meulemans and Johannes von Oswald and Benjamin F. Grewe},
  booktitle={International Conference on Learning Representations},
  year={2021},
  url={https://arxiv.org/abs/2006.12109}
}

@inproceedings{posterior:replay:2021:henning:cervera,
title={Posterior Meta-Replay for Continual Learning}, 
      author={Christian Henning and Maria R. Cervera and Francesco D'Angelo and Johannes von Oswald and Regina Traber and Benjamin Ehret and Seijin Kobayashi and João Sacramento and Benjamin F. Grewe},
booktitle={Conference on Neural Information Processing Systems},
year={2021},
url={https://arxiv.org/abs/2103.01133}
}
Comments
  • How to have a single hypernetwork for generating two main networks with different architectures?

    How to have a single hypernetwork for generating two main networks with different architectures?

    Hi, I'm fairly new to Hypernetworks, so not sure if this would be a reasonable approach. But is there a way to have a single hypernet with a conditional input, that depending on the condition, generates weights for different architectures. The architectures are not very different, for example both are MLPs, where one would have an n-dimensional output whereas the other has an m-dimensional output.

    One way I could see it implemented is to have the hypernetwork create a list of weights but then manually pick the corresponding weights of each architecture. But then this may not be that different from training two independent hypernets and so the learning may not be factorized. Would there be a better and smarter approach based on the current functionalities of the package?

    Thanks a lot! Sahand

    opened by sahandrez 2
  • Forward pass of the main network with a batch of weights

    Forward pass of the main network with a batch of weights

    Hi,

    In the context of MLPs, I know that the forward pass of the hypernetwork can produce a batch of weights for the main network. But it seems that the forward pass of the main network is not able to process the batch of weights due to this assertion error:

    File "/Users/sahand.r/workspace/hypnettorch/hypnettorch/mnets/mlp.py", line 561, in forward
    assert(len(weights) == len(self.param_shapes))
AssertionError

    A naive approach would be to loop over the batch and do independent forward passes of the main network, but is there a more efficient approach for this?

    Thanks!

    opened by sahandrez 2
  • Training for MNIST and FashionMNIST datasets.

    Training for MNIST and FashionMNIST datasets.

    opened by ZS123-lang 2
  • Hypernetwork for Fourier neural operators

    Hypernetwork for Fourier neural operators

    Hello developers,

    Thank you for providing this library. I would like to know if you have plans to develop similar method for fourier neural operator as main network?

    Fourier neural operators cannot take meta data as input and in a way it can be incorporated by making the Weight matrix as hyperparameter. Do you think u can look into it.

    Thanks

    opened by Tariq97 1
  • Incorrect dtype of weights tensor in getting_started tutorial

    Incorrect dtype of weights tensor in getting_started tutorial

    Hi, First of all thanks for the great work on this library, it really helped me with my own research work. I noticed small bug in getting_started tutorial. If device is initally set to cuda, then later on when hypernetwork is used to generate main net weights this error occurs due to mismatch between types of image tensor X placed on cuda and weights tensor W which is placed on cpu. image Simple fix is to move hypernetwork to device like this:

    hnet = HMLP(mnet.param_shapes, uncond_in_size=0, cond_in_size=8,
            layers=[100, 100], num_cond_embs=2).to(device)

    Then it should work regardless if device is set to 'cpu' or 'cuda'.

    opened by fszatkowski 1
  • Graph Neural Network as MainNet

    Graph Neural Network as MainNet

    Hi, I was wondering if there is a way to use other types of main net not implemented in this library? For example, would it be possible to use a PyTorch Geometric network as the main net?

    opened by yhl48 1
Owner
Christian Henning
Machine Learning Researcher
Christian Henning
GitHub
working repo for my xumx-sliCQ submissions to the ISMIR 2021 MDX

Music Demixing Challenge - xumx-sliCQ This repository is the GitHub mirror of my working submission repository for the AICrowd ISMIR 2021 Music Demixi

null 4 Aug 25, 2021
A set of tools for converting a darknet dataset to COCO format working with YOLOX

darknet格式数据→COCO darknet训练数据目录结构（详情参见dataset/darknet）： darknet ├── class.names ├── gen_config.data ├── gen_train.txt ├── gen_valid.txt └── images

RapidAI-NG 148 Jan 3, 2023
Quantile Regression DQN a Minimal Working Example, Distributional Reinforcement Learning with Quantile Regression

Quantile Regression DQN Quantile Regression DQN a Minimal Working Example, Distributional Reinforcement Learning with Quantile Regression (https://arx

Arsenii Senya Ashukha 80 Sep 17, 2022
Tool for working with Y-chromosome data from YFull and FTDNA

ycomp ycomp is a tool for working with Y-chromosome data from YFull and FTDNA. Run ycomp -h for information on how to use the program. Installation Th

Alexander Regueiro 2 Jun 18, 2022
MacroTools provides a library of tools for working with Julia code and expressions.

MacroTools.jl MacroTools provides a library of tools for working with Julia code and expressions. This includes a powerful template-matching system an

FluxML 278 Dec 11, 2022
Working demo of the Multi-class and Anomaly classification model using the CLIP feature space

??️ Hindsight AI: Crime Classification With Clip About For Educational Purposes Only This is a recursive neural net trained to classify specific crime

Miles Tweed 2 Jun 5, 2022
Addon and nodes for working with structural biology and molecular data in Blender.

Molecular Nodes ?? ?? ?? Buy Me a Coffee to Keep Development Going! Join a Community of Blender SciVis People! What is Molecular Nodes? Molecular Node

Brady Johnston 456 Jan 8, 2023
Implementation of trRosetta and trDesign for Pytorch, made into a convenient package

trRosetta - Pytorch (wip) Implementation of trRosetta and trDesign for Pytorch, made into a convenient package

Phil Wang 67 Dec 17, 2022
Kindle is an easy model build package for PyTorch.

Kindle is an easy model build package for PyTorch. Building a deep learning model became so simple that almost all model can be made by copy and paste from other existing model codes. So why code? when we can simply build a model with yaml markup file. Kindle builds a model with no code but yaml file which its method is inspired from YOLOv5.

Jongkuk Lim 77 Nov 11, 2022
PyTorch package for the discrete VAE used for DALL·E.

Overview [Blog] [Paper] [Model Card] [Usage] This is the official PyTorch package for the discrete VAE used for DALL·E. Installation Before running th

OpenAI 9.5k Jan 5, 2023
Python package facilitating the use of Bayesian Deep Learning methods with Variational Inference for PyTorch

PyVarInf PyVarInf provides facilities to easily train your PyTorch neural network models using variational inference. Bayesian Deep Learning with Vari

null 342 Dec 2, 2022
tsai is an open-source deep learning package built on top of Pytorch & fastai focused on state-of-the-art techniques for time series classification, regression and forecasting.

Time series Timeseries Deep Learning Pytorch fastai - State-of-the-art Deep Learning with Time Series and Sequences in Pytorch / fastai

timeseriesAI 2.8k Jan 8, 2023
quantize aware training package for NCNN on pytorch

ncnnqat ncnnqat is a quantize aware training package for NCNN on pytorch. Table of Contents ncnnqat Table of Contents Installation Usage Code Examples

null 62 Nov 23, 2022
Implementation of the Triangle Multiplicative module, used in Alphafold2 as an efficient way to mix rows or columns of a 2d feature map, as a standalone package for Pytorch

Triangle Multiplicative Module - Pytorch Implementation of the Triangle Multiplicative module, used in Alphafold2 as an efficient way to mix rows or c

Phil Wang 22 Oct 28, 2022
Leibniz is a python package which provide facilities to express learnable partial differential equations with PyTorch

Leibniz is a python package which provide facilities to express learnable partial differential equations with PyTorch

Beijing ColorfulClouds Technology Co.,Ltd. 16 Aug 7, 2022
This package proposes simplified exporting pytorch models to ONNX and TensorRT, and also gives some base interface for model inference.

PyTorch Infer Utils This package proposes simplified exporting pytorch models to ONNX and TensorRT, and also gives some base interface for model infer

Alex Gorodnitskiy 11 Mar 20, 2022
The hippynn python package - a modular library for atomistic machine learning with pytorch.

The hippynn python package - a modular library for atomistic machine learning with pytorch. We aim to provide a powerful library for the training of a

Los Alamos National Laboratory 37 Dec 29, 2022
Pytorch domain adaptation package

DomainAdaptation This package is created to tackle the problem of domain shifts when dealing with two domains of different feature distributions. In d

Institute of Computational Perception 7 Oct 22, 2022
Non-stationary GP package written from scratch in PyTorch

NSGP-Torch Examples gpytorch model with skgpytorch # Import packages import torch from regdata import NonStat2D from gpytorch.kernels import RBFKernel

Zeel B Patel 1 Mar 6, 2022