A PyTorch implementation of Learning to learn by gradient descent by gradient descent

Overview
Comments
  • Wrong log_softmax dimension in model

    Wrong log_softmax dimension in model

    The following line computes the log_softmax over the batch dimension (default: dim=0): https://github.com/ikostrikov/pytorch-meta-optimizer/blob/0154d4d4fc856163885a62bac06174311aa58caf/model.py#L19

    This should include a dim=-1 to result in combination with F.nll_loss into the cross entropy loss.

    opened by timmeinhardt 1
  • Why do you input grad & params into metaoptimizer?

    Why do you input grad & params into metaoptimizer?

    In line 81 & 135 of meta_optimizer.py Why do you input grads & params (and loss as well in line 135) into metaoptimizer?

    Did you try with & without, and one version worked better? In 1606.04474, they seem to just input grads into metaoptimizer.

    opened by ethancaballero 1
  • FastMetaLearner equal to Optimization as a model for few-shot learning

    FastMetaLearner equal to Optimization as a model for few-shot learning

    Hi, your implementation on the whole model is really cool! I learned a lot from reading your code.

    And I want to know whether the FastMetaLearner class you implement is just equal to the paper Optimization as a model for few-shot learning?

    opened by yuhui-zh15 0
  • pytorch 0.3 correction

    pytorch 0.3 correction

    The current version of the code produce a NxN matrix of weight instead of a N vector of weight (N the number of weight of the model to optimize). I modify a bit the code to produce the correct vector.

    opened by Forbu 0
  • Might work better with a different meta-optimizer - enhancement

    Might work better with a different meta-optimizer - enhancement

    opened by AjayTalati 0
  • Which article does Fast-Meta-Optimizer come from?

    Which article does Fast-Meta-Optimizer come from?

    Thanks for your code, i learn a lot from it. But i can't find the source of Fast-Meta-Optimizer. It looks like meta-SGD, but not common entirely. Please tell me the source of this method, thanks!

    opened by ghost 1
  • Visualize the best LSTM loss

    Visualize the best LSTM loss

    Hi,

    I am new to meta-learning and trying to understand your code. How do I get a figure of the LSTM loss-steps like the author showed in the paper if I use matplotlib?

    Shall I catch the best final_loss as y-axis and use optimizer_steps as x-axis?

    opened by NaV1YChen 0
  • LSTM weights not optimised

    LSTM weights not optimised

    The LayerNormLSTMCell modules initialised in the MetaOptimizer class are not properly registered as parameters of the MetaOptimizer model. Appending them to the self.lstms list: https://github.com/ikostrikov/pytorch-meta-optimizer/blob/0154d4d4fc856163885a62bac06174311aa58caf/meta_optimizer.py#L27

    will not add their trainable parameters to the model parameter list in:

    https://github.com/ikostrikov/pytorch-meta-optimizer/blob/0154d4d4fc856163885a62bac06174311aa58caf/main.py#L63

    If I am not mistaken, the current version will not train the LSTM weights at all. In general, I would suggest to restructure the initialisation and MetaOptimizer.forward method, but as a quick fix one could replace the entire self.lstms initialization block with this:

    self.lstms = nn.Sequential(*[LayerNormLSTMCell(hidden_size, hidden_size)
                                                    for _ in range(num_layers)])
    
    opened by timmeinhardt 3
  • the meaning of preprocess_gradients function

    the meaning of preprocess_gradients function

    Hi, I've learned so much from your code, but stiil have a small question. What does the function 'preprocess_gradients ' do in utils.py? Could you please explain it?

    opened by littleso-so 0
  • Error when running main.py

    Error when running main.py

    Hi, I am using pytorch 0.2 and when I run the script, it generated following errors:

    Traceback (most recent call last):
      File "main.py", line 142, in <module>
        main()
      File "main.py", line 116, in main
        meta_model = meta_optimizer.meta_update(model, loss.data)
      File "/home/rvl224/pytorch-meta-optimizer/meta_optimizer.py", line 135, in meta_update
        inputs = Variable(torch.cat((preprocess_gradients(flat_grads), flat_params.data, loss), 1))
      File "/home/rvl224/pytorch-meta-optimizer/utils.py", line 11, in preprocess_gradients
        return torch.cat((x1, x2), 1)
    RuntimeError: dim out of range - got 1 but the tensor is only 1D
    
    

    Is it because I am using a different version of pytorch?

    opened by ghost 5
  • Out of memory when the meta optimizer updates parameters

    Out of memory when the meta optimizer updates parameters

    Hello, I find your code very helpful, but too much memory is consumed when the meta optimizer updates parameters of the model. On my computer, it always raises an error 'out of memory' when executes Line 140 of meta_optimizer.py.

    I think it could consume less memory if the MetaModel class holds a flat version of parameters instead of wrapping a model. In this way, the MetaModel reshapes the parameters and computes result through nn.functional.conv/linear, so that the meta optimizer can directly use this flat version of parameters, without allocating extra memory for flatted parameters.

    opened by zengxianyu 3
Owner
Ilya Kostrikov
Post doc
Ilya Kostrikov
Plotting points that lie on the intersection of the given curves using gradient descent.

Plotting intersection of curves using gradient descent Webapp Link ---> What's the app about Why this app Plotting functions and their intersection. A

Divakar Verma 2 Jan 9, 2022
Fibonacci Method Gradient Descent

An implementation of the Fibonacci method for gradient descent, featuring a TKinter GUI for inputting the function / parameters to be examined and a matplotlib plot of the function and results.

Emma 1 Jan 28, 2022
This project uses reinforcement learning on stock market and agent tries to learn trading. The goal is to check if the agent can learn to read tape. The project is dedicated to hero in life great Jesse Livermore.

Reinforcement-trading This project uses Reinforcement learning on stock market and agent tries to learn trading. The goal is to check if the agent can

Deepender Singla 1.4k Dec 22, 2022
Objective of the repository is to learn and build machine learning models using Pytorch. 30DaysofML Using Pytorch

30 Days Of Machine Learning Using Pytorch Objective of the repository is to learn and build machine learning models using Pytorch. List of Algorithms

Mayur 119 Nov 24, 2022
Pytorch Implementation of paper "Noisy Natural Gradient as Variational Inference"

Noisy Natural Gradient as Variational Inference PyTorch implementation of Noisy Natural Gradient as Variational Inference. Requirements Python 3 Pytor

Tony JiHyun Kim 119 Dec 2, 2022
PyTorch implementation of Spiking Neural Networks trained on surrogate gradient & BPTT using snntorch.

snn-localization repo PyTorch implementation of Spiking Neural Networks trained on surrogate gradient & BPTT using snntorch. Install Dependencies Orig

Sami BARCHID 1 Jan 6, 2022
Storchastic is a PyTorch library for stochastic gradient estimation in Deep Learning

Storchastic is a PyTorch library for stochastic gradient estimation in Deep Learning

Emile van Krieken 140 Dec 30, 2022
Official implementation of "GS-WGAN: A Gradient-Sanitized Approach for Learning Differentially Private Generators" (NeurIPS 2020)

GS-WGAN This repository contains the implementation for GS-WGAN: A Gradient-Sanitized Approach for Learning Differentially Private Generators (NeurIPS

null 46 Nov 9, 2022
Implementation of Learning Gradient Fields for Molecular Conformation Generation (ICML 2021).

[PDF] | [Slides] The official implementation of Learning Gradient Fields for Molecular Conformation Generation (ICML 2021 Long talk) Installation Inst

MilaGraph 117 Dec 9, 2022
Pytorch implementation for "Implicit Feature Alignment: Learn to Convert Text Recognizer to Text Spotter".

Implicit Feature Alignment: Learn to Convert Text Recognizer to Text Spotter This is a pytorch-based implementation for paper Implicit Feature Alignme

wangtianwei 61 Nov 12, 2022
PyTorch implementation of "Learn to Dance with AIST++: Music Conditioned 3D Dance Generation."

Learn to Dance with AIST++: Music Conditioned 3D Dance Generation. Installation pip install -r requirements.txt Prepare Dataset bash data/scripts/pre

Zj Li 8 Sep 7, 2021
An original implementation of "MetaICL Learning to Learn In Context" by Sewon Min, Mike Lewis, Luke Zettlemoyer and Hannaneh Hajishirzi

MetaICL: Learning to Learn In Context This includes an original implementation of "MetaICL: Learning to Learn In Context" by Sewon Min, Mike Lewis, Lu

Meta Research 27 Nov 2, 2021
NFNets and Adaptive Gradient Clipping for SGD implemented in PyTorch

PyTorch implementation of Normalizer-Free Networks and SGD - Adaptive Gradient Clipping Paper: https://arxiv.org/abs/2102.06171.pdf Original code: htt

Vaibhav Balloli 320 Jan 2, 2023
Keras implementation of Normalizer-Free Networks and SGD - Adaptive Gradient Clipping

Keras implementation of Normalizer-Free Networks and SGD - Adaptive Gradient Clipping

Yam Peleg 63 Sep 21, 2022
The official implementation of You Only Compress Once: Towards Effective and Elastic BERT Compression via Exploit-Explore Stochastic Nature Gradient.

You Only Compress Once: Towards Effective and Elastic BERT Compression via Exploit-Explore Stochastic Nature Gradient (paper) @misc{zhang2021compress,

null 46 Dec 7, 2022
Official implementation of paper Gradient Matching for Domain Generalization

Gradient Matching for Domain Generalisation This is the official PyTorch implementation of Gradient Matching for Domain Generalisation. In our paper,

null 94 Dec 23, 2022
Official implementation of Generalized Data Weighting via Class-level Gradient Manipulation (NeurIPS 2021).

Generalized Data Weighting via Class-level Gradient Manipulation This repository is the official implementation of Generalized Data Weighting via Clas

null 9 Nov 3, 2021
A fast, scalable, high performance Gradient Boosting on Decision Trees library, used for ranking, classification, regression and other machine learning tasks for Python, R, Java, C++. Supports computation on CPU and GPU.

Website | Documentation | Tutorials | Installation | Release Notes CatBoost is a machine learning method based on gradient boosting over decision tree

CatBoost 6.9k Jan 4, 2023