• implement MinibatchIterator class - a reusable iterator over minibatches
• return an instance of MinibatchIterator from minibatches fun
• the new iterator creates 1 minibatch at a time saving GPU memory

I just realized that if we never calculate more than needed in the optimizers loop (because it can be done from the outside) we actually don't need the stop functionality. Yields are rather fast (compared to model evaluations). This would make code a lot easier.

Any objections?

Several reasons might lead you to stop optimization:

1. the gradient is 0,
2. the change of the parameters is negligible,
3. a finite amout of time has passed,
4. a desired error has been reached,
5. a finite amount of function/gradient evaluations has been done,
6. a finite amount of iterations has been done.

It would be nice to have convenience functions for this. Most are easy, but e.g. 2. needs to keep track of values--thus, the stopping criterion is stateful. I have a feeling we will overshoot if we solve all of these.

The tests in test_rprop.py fail in the master version. I've verified that the failure is unrelate to the compatibility fixes.

Also, it seems the builds on Travis CI fail with timeout. I can alter .travis.yml to use conda for NumPy/SciPy. This should significantly speed up the installation step. What do you think?

introducing an initial step rate as recommended for irprop- fixes the slow start. in some cases it was barely usable, as getting from min_step to a decent step rate takes forever. i also removed the changes_max parameter as it was never used.

@bayerj It seems there is a bug in adadelta.py when momentum is used. The momentum correction can be applied to adadelta, rmrprop and others stochastic updates. The potential bug is at the 110-th line of adadelta.py

    def _iterate(self):
        for args, kwargs in self.args:
            step_m1 = self.step
            d = self.decay
            o = self.offset
            m = self.momentum
            step1 = step_m1 * m * self.step_rate
            self.wrt -= step1

            gradient = self.fprime(self.wrt, *args, **kwargs)

            self.gms = (d * self.gms) + (1 - d) * gradient ** 2
            step2 = sqrt(self.sms + o) / sqrt(self.gms + o) * gradient * self.step_rate
            self.wrt -= step2

            self.step = step1 + step2
            self.sms = (d * self.sms) + (1 - d) * self.step ** 2

            self.n_iter += 1

            yield {
                'n_iter': self.n_iter,
                'gradient': gradient,
                'args': args,
                'kwargs': kwargs,
            }

I think it should be step1 = step_m1 * m instead of step1 = step_m1 * m * self.step_rate. Correct me if I am wrong.

Note that in rmsprop.py, the 160-th line is

 step1 = step_m1 * self.momentum

, which is correct.

Let me point out that with anaconda 2.5, python 2.7 and windows x64, after installing climin with pip, it is necessary to comment out this section in init.py of climin.

if sys.platform == 'win32':

basepath = imp.find_module('numpy')[1]

ctypes.CDLL(os.path.join(basepath, 'core', 'libmmd.dll'))

ctypes.CDLL(os.path.join(basepath, 'core', 'libifcoremd.dll'))

then climin runs superbly.

Today 64-bit machines are probably the majority and 32-bit machines an exception.

There is an issue with FORTAN libraries replacing signal handlers and not being able to recover them:

http://stackoverflow.com/questions/15457786/ctrl-c-crashes-python-after-importing-scipy-stats

The solution is to

Add to climin/__init__.py:

if sys.platform == 'win32':
  basepath = imp.find_module('numpy')[1]
  ctypes.CDLL(os.path.join(basepath, 'core', 'libmmd.dll'))
  ctypes.CDLL(os.path.join(basepath, 'core', 'libifcoremd.dll'))

And then extend OnSignal with

import win32api
win32api.SetConsoleCtrlHandler(self._console_ctrl_handler, 1)

in the windows case.

After that, climin has to be imported before scipy by the user.

previously, TimeElapsed would simply calculate time from its instantiation, which is problematic if the experiment is interrupted thereafter. Now it can read out the info dict which can hold a more meaningful runtime value.

The decay parameter is no longer included in the original adam paper anymore (was apparently only needed for a convergence proof), thus it is set to None and a warning is issued if the user deliberately sets it to another value. Also fixes a bug in the (nesterov) momentum step and includes an optimization mentioned in the adam paper.

A newer Anaconda version (tested with 4.1.5) comes with numpy version 1.10.4 which uses newer Fortran compilers. The old fix fails, but the newly introduced flag 'FOR_DISABLE_CONSOLE_CTRL_HANDLER' (Intel Fortran version >=16.0) can be utilized instead for preventing breaking of Ctrl+C/Break.

Optimization schemes with complex numbers are widely used in physics, and recently, machine learning.

I strongly suggest to add the support to complex numbers for optimization engines like RmsProp et. al.

We just need a few lines of change and several tests.

E.g. climin/rmsprop.py line 165-167

            self.moving_mean_squared = (
                self.decay * self.moving_mean_squared
                + (1 - self.decay) * gradient ** 2) 
            --> + (1 - self.decay) * np.abs(gradient) ** 2)

A single line of change would make it applicable for complex numbers.

The same is true for Adam and Adadelta.

On the other side, GradientDescent works well already without any change.

Maybe a bit effort is needed for Rprop, I have no clue yet how to make it compatible with complex numbers due to the ill defined sign function for complex numbers.

Contrary to what the docstring says, gradient descent does not accept a sequence for the step_rate parameter.

I'm happy to submit a pull request for this, if there's still interest!

Hi, I'd love to see this package in PyPI. Any plans to do so? It would help installing the package and declaring it as a dependency.

Just to reserve the package name and to test if things work, I registered and uploaded the most recent version to PyPI. I will either remove the package from PyPI or move the PyPI package ownership to you, whatever you wish.

In order to use PyPI, you'd need to fix version numbering to follow PEP 0440: https://www.python.org/dev/peps/pep-0440/. Thus, you would need to change the version number in setup.py to something like 0.1a1, 0.1b4, 0.1rc2 etc.

Uploading to PyPI can be done as python setup.py sdist upload.

If you are interested in this and need any help, I'd be happy to help if I can.

@bayerj The 152-th line of rmsprop.py

        self.moving_mean_squared = 1

I think the initial value should be 0 instead of 1. Any reason why 1 is better than 0