The following line computes the log_softmax over the batch dimension (default: dim=0): https://github.com/ikostrikov/pytorch-meta-optimizer/blob/0154d4d4fc856163885a62bac06174311aa58caf/model.py#L19

This should include a dim=-1 to result in combination with F.nll_loss into the cross entropy loss.

In line 81 & 135 of meta_optimizer.py Why do you input grads & params (and loss as well in line 135) into metaoptimizer?

Did you try with & without, and one version worked better? In 1606.04474, they seem to just input grads into metaoptimizer.

Hi, your implementation on the whole model is really cool! I learned a lot from reading your code.

And I want to know whether the FastMetaLearner class you implement is just equal to the paper Optimization as a model for few-shot learning?

The current version of the code produce a NxN matrix of weight instead of a N vector of weight (N the number of weight of the model to optimize). I modify a bit the code to produce the correct vector.

Hi @ikostrikov,

I just saw this, Learning Gradient Descent: Better Generalization and Longer Horizons, and their TF repo, rnnprop

They claim their meta-optimizer works better than deepminds? Just thought it's worth posting here, in case anyone want's to look into it, as an enhancement to this repo?

Cheers,

Aj

Thanks for your code, i learn a lot from it. But i can't find the source of Fast-Meta-Optimizer. It looks like meta-SGD, but not common entirely. Please tell me the source of this method, thanks!

Hi,

I am new to meta-learning and trying to understand your code. How do I get a figure of the LSTM loss-steps like the author showed in the paper if I use matplotlib?

Shall I catch the best final_loss as y-axis and use optimizer_steps as x-axis?

The LayerNormLSTMCell modules initialised in the MetaOptimizer class are not properly registered as parameters of the MetaOptimizer model. Appending them to the self.lstms list: https://github.com/ikostrikov/pytorch-meta-optimizer/blob/0154d4d4fc856163885a62bac06174311aa58caf/meta_optimizer.py#L27

will not add their trainable parameters to the model parameter list in:

https://github.com/ikostrikov/pytorch-meta-optimizer/blob/0154d4d4fc856163885a62bac06174311aa58caf/main.py#L63

If I am not mistaken, the current version will not train the LSTM weights at all. In general, I would suggest to restructure the initialisation and MetaOptimizer.forward method, but as a quick fix one could replace the entire self.lstms initialization block with this:

self.lstms = nn.Sequential(*[LayerNormLSTMCell(hidden_size, hidden_size)
                                                for _ in range(num_layers)])

Hi, I've learned so much from your code, but stiil have a small question. What does the function 'preprocess_gradients ' do in utils.py? Could you please explain it?

Hi, I am using pytorch 0.2 and when I run the script, it generated following errors:

Traceback (most recent call last):
  File "main.py", line 142, in <module>
    main()
  File "main.py", line 116, in main
    meta_model = meta_optimizer.meta_update(model, loss.data)
  File "/home/rvl224/pytorch-meta-optimizer/meta_optimizer.py", line 135, in meta_update
    inputs = Variable(torch.cat((preprocess_gradients(flat_grads), flat_params.data, loss), 1))
  File "/home/rvl224/pytorch-meta-optimizer/utils.py", line 11, in preprocess_gradients
    return torch.cat((x1, x2), 1)
RuntimeError: dim out of range - got 1 but the tensor is only 1D

Is it because I am using a different version of pytorch?

Hello, I find your code very helpful, but too much memory is consumed when the meta optimizer updates parameters of the model. On my computer, it always raises an error 'out of memory' when executes Line 140 of meta_optimizer.py.

I think it could consume less memory if the MetaModel class holds a flat version of parameters instead of wrapping a model. In this way, the MetaModel reshapes the parameters and computes result through nn.functional.conv/linear, so that the meta optimizer can directly use this flat version of parameters, without allocating extra memory for flatted parameters.