Thanks for open sourcing the code !

I've tried it on a simple MLP and could not find a set of parameters (lr and mu) that would not yield one of those two errors:

assert root.size == 1 AssertionError

and

numpy.linalg.linalg.LinAlgError: Array must not contain infs or NaNs

Any tips on best practices ?

Hi,

I'm running into a situation where YellowFin keeps adjusting the LR upward instead of decaying it downward - which, of course, prevents the network from converging. Any idea why it would be happening? Thanks!

I initialize YF like so: optimizer = YFOptimizer(net.parameters(), lr=train_args['lr'])

And the output is...

[epoch 4], [iter 20 / 123], [train main loss 0.15482], [lr 0.057262]
[epoch 4], [iter 40 / 123], [train main loss 0.14807], [lr 0.058468]
[epoch 4], [iter 60 / 123], [train main loss 0.14976], [lr 0.059635]
[epoch 4], [iter 80 / 123], [train main loss 0.14867], [lr 0.060768]
[epoch 4], [iter 100 / 123], [train main loss 0.14935], [lr 0.061881]
[epoch 4], [iter 120 / 123], [train main loss 0.14653], [lr 0.062980]

----------------------------------------------------------------------------------------------
[epoch 5], [iter 20 / 123], [train main loss 0.14709], [lr 0.064228]
[epoch 5], [iter 40 / 123], [train main loss 0.14188], [lr 0.065275]
[epoch 5], [iter 60 / 123], [train main loss 0.16187], [lr 0.066297]
[epoch 5], [iter 80 / 123], [train main loss 0.15231], [lr 0.067289]
[epoch 5], [iter 100 / 123], [train main loss 0.15639], [lr 0.068227]
[epoch 5], [iter 120 / 123], [train main loss 0.15515], [lr 0.069117]

----------------------------------------------------------------------------------------------
[epoch 6], [iter 20 / 123], [train main loss 0.13752], [lr 0.070135]
[epoch 6], [iter 40 / 123], [train main loss 0.13210], [lr 0.071002]
[epoch 6], [iter 60 / 123], [train main loss 0.13821], [lr 0.071850]
[epoch 6], [iter 80 / 123], [train main loss 0.13456], [lr 0.072690]
[epoch 6], [iter 100 / 123], [train main loss 0.13225], [lr 0.073533]
[epoch 6], [iter 120 / 123], [train main loss 0.13367], [lr 0.074379]

Hi, I tried your optimizer instead of SGD for this challenge https://www.kaggle.com/c/planet-understanding-the-amazon-from-space and get this kind of train/valid curves https://s.mail.ru/BFbQ/1K6w1bqD7 , which is obv awful (tried different setups but no success). SGD & plateau scheduler reach 0.08258 val loss.

What data do you need to investigate such bad performance ? For example learning rates changes between 0.1 and 3 (!) .

Hi,

The following error pops up when I use yellowfin.py in my setup.

File "/home/bchatter/Documents/workspace/python/async-opt/resnet_cifar_yellowfin/yellowfin.py", line 568, in step
    self.load_state_dict_perturb(copy.deepcopy(self._state_checkpoint) )
AttributeError: 'YFOptimizer' object has no attribute '_state_checkpoint'

I attempted solving this issue by copying the line 543 in yellowfin.py self._state_checkpoint = copy.deepcopy(self.state_dict() ) just above the line 568. However, with that the optimizer starts to be non-converging (at times diverges, but certainly does not converge).

Could you please look at the issue?

Hi,

For YellowFin optimzer, do I need to use the learning rate decay trick? For example, I evaluate the model on dev set and if the performance drops I will halve the learning rate. This trick works very well for optimizers such as Adam and Adadelta. So will it also work if I switch to the YellowFin optimizer?

Thanks.

Feel free to close since it seems like a PyTorch bug, but as a heads up, in case others hit the same issue, on some runs I got this:

Traceback (most recent call last):
  File "train.py", line 219, in <module>
    optimizer.step()
  File "/home/grant/repos/aud1/yellowfin.py", line 371, in step
    self.after_apply()
  File "/home/grant/repos/aud1/yellowfin.py", line 276, in after_apply
    self.grad_sparsity()
  File "/home/grant/repos/aud1/yellowfin.py", line 219, in grad_sparsity
    grad_non_zero = grad.nonzero()
RuntimeError: an illegal memory access was encountered

Trying to address: https://github.com/JianGoForIt/YellowFin_Pytorch/issues/9

The approach is to save anything that is potentially "stateful" in the optimizer. A quick sanity check indicates that this works.

Sanity check: https://gist.github.com/mrdrozdov/d3d6ac43a3130c799e4f3f5867853184

thanks for this code release, I wish more papers did this! It really makes it effortless to try out this new optimizer :)

It'd be great if YellowFin supported the state_dict() and load_state_dict() functions, to maintain a consistent serialization API with the other PyTorch optimizers.

In the publication the variance in "Algorithm 3 Gradient variance" is defined as: However in the PyTorch implementation variance is defined as: Did you try YellowFin with variance from the publication? Were the results worse? Which definition of the variance did you use to produce results from the publication?

Hi! Thank you for your great work!

Could you tell why you decided to use one global alpha and one global mu for whole the model instead of creating a separate alpha and mu for each matrix of weights in the model? The other approach seems to be more natural to me because each matrix of weights might have different distributions of its values and values of its gradient. Did you consider it? Do you see a reason not to do so?

I get the following error with the following stack: optimizer.step() self.after_apply() self.get_mu() assert root.size == 1

It works fine for most of my runs but fails the assertion sometimes. I will try reproducing it in a simpler setting.

There seems two be two reasons for this:

1. 0.4 introduced 0-dimensional tensors (scalars) and to get their value as a python float we need to call .item() on them. If we don't (and train on GPU) yellowfin will hold on to tensors on both CPU and GPU and try to do operations on them (which will cause an exception since they are on different devices, that exception will be swallowed by the checkpoint restoration mechanism).
2. Tensors and Variables have been merged in 0.4, so unless the code is changed yellowfin will hold on to tensors with gradient history causing a memory leak.

The first issue seems to be quite easy to patch, I can send a pull-request for that part if you want to.

Hello there,

I am doing my best to learn how to use this optimizer, as I would very much like to have an auto-tuned optimizer where I do not have to spend endless days fiddling with hyperparameters. I have tried to use YellowFin to learn large vision models such as MobileNet, but my results are always very disappointing as compared to a traditional optimizer such as SGD. I am not so concerned about convergence time as I am about loss/accuracy; I have found that YellowFin tends to converge to a much worse loss/accuracy than my SGD runs do.

I am posting here an example of training MobileNet on the ImageNet dataset with a batch size of 64, comparing the training and testing loss (as well as testing accuracy) of a few epochs of training on MobileNet. In both cases, I have a learning rate schedule applied to set the learning rate factor to 0.3 ^ (epoch // 10), which causes the learning rate to fall to 3/10 of its value every 10 epochs. You can see the effect of this learning rate schedule in the sgd plot fairly easily, the yf plot shows it less clearly. In these figures, the training loss (per minibatch) is shown in blue, while the testing loss (per epoch) is shown in red, with the relevant axis shown on the left. The top-1 and top-5 accuracies on the training dataset are shown in green (per epoch), with their relevant axis given on the right. Other than the optimizer choice, all other training settings are the same, including minibatch size (64), dataset (ImageNet) and model architecture (MobileNet).

Here is a plot for an SGD optimizer run (note that I have this model only partially trained, this is because it has trained enough that we can already see it will converge to a significantly better loss than the YF model did, below):

Here is a plot for a YellowFin optimizer run:

If there are any questions about my methodology I would be happy to explain in greater detail. There is nothing particularly special going on in my model, I am simply trying to determine why YellowFin seems to converge with such poor results.

When I try to save the optimizer.state_dict() before the first training step, this error occurred. Seems that we should add self._h_min = 0.0 and self._h_max = 0.0 in __init__() ?

I was trying to experiment with the effects of changing the clipping threshold during training, but I noticed that my changes were getting overridden because this is kept as state. Also some other user set options are kept in the state, but probably should not be lest it be difficult to change them during training.

When resuming from a checkpoint, self.iter is not zero, but self._h_max is still undefined, so the optimizer would error. The PR checks directly if self._h_max is defined, which should always work.