Using PyCudaHandler, I get:

[Traceback (most recent call last):
  File "mnist_lstm.py", line 85, in <module>
    trainer.train(network, train_getter, valid_getter=valid_getter)
  File "build/bdist.linux-x86_64/egg/brainstorm/training/trainer.py", line 50, in train
  File "build/bdist.linux-x86_64/egg/brainstorm/training/trainer.py", line 126, in _emit_monitoring
  File "build/bdist.linux-x86_64/egg/brainstorm/training/trainer.py", line 136, in _call_monitor
  File "build/bdist.linux-x86_64/egg/brainstorm/training/monitors.py", line 289, in __call__
  File "build/bdist.linux-x86_64/egg/brainstorm/training/trainer.py", line 169, in run_network
  File "build/bdist.linux-x86_64/egg/brainstorm/structure/network.py", line 62, in provide_external_data
  File "build/bdist.linux-x86_64/egg/brainstorm/handlers/pycuda_handler.py", line 49, in copy_to
pycuda._driver.LogicError: cuMemcpyDtoD failed: invalid argument in <brainstorm.training.monitors.MonitorAccuracy object at 0x7fea2cf61050>

@untom, perhaps you have an idea about what might be causing this?

We just briefly discussed this and I think it would be good to change the buffer layout for our convolutional layers from 1 to 2:

1. (time, batch_size, channels, height, width)
2. (time, batch_size, height, width, channels)

Pros

• Batch-normalization for conv layers would be easier, because there you want to estimate the mean/std over all time, batch_size, height, and width but not channels. And that you could then do with a single flatten operation.
• When writing an 2D-RNN or 2D-LSTM implementation, it would then probably be possible to do the matrix multiplication for the recurrent matrices with a single BLAS call over all images in the batch, because only a single striding is needed, and not two.
• It makes more sense from a memory layout perspective, because each filter of a convolution uses all channels but not the entire width/height.
• OpenCV and apparently caffe2 use that format

Cons

• Caffe uses format 1
• We need to change our code :-(

Addition of convolution/pooling layers will significantly add capabilities to the library. The base has been designed keeping this in mind, so having these is a pre-alpha goal.

The recent added functionality seems to have broken the PyCudaHandler (including the tests). Running the tests results in:

Traceback (most recent call last):
  File "/home/arkade/venv/py2/local/lib/python2.7/site-packages/_pytest/config.py", line 543, in importconftest
    mod = conftestpath.pyimport()
  File "/home/arkade/venv/py2/local/lib/python2.7/site-packages/py/_path/local.py", line 650, in pyimport
    __import__(modname)
  File "/home/arkade/Dropbox/codes/brainstorm/test/conftest.py", line 7, in <module>
    from brainstorm.structure.architecture import (
  File "/home/arkade/Dropbox/codes/brainstorm/brainstorm/__init__.py", line 5, in <module>
    from brainstorm.structure import *
  File "/home/arkade/Dropbox/codes/brainstorm/brainstorm/structure/__init__.py", line 4, in <module>
    from brainstorm.structure.network import Network
  File "/home/arkade/Dropbox/codes/brainstorm/brainstorm/structure/network.py", line 11, in <module>
    from brainstorm.structure.buffers import BufferManager
  File "/home/arkade/Dropbox/codes/brainstorm/brainstorm/structure/buffers.py", line 6, in <module>
    from brainstorm.handlers import default_handler
  File "/home/arkade/Dropbox/codes/brainstorm/brainstorm/handlers/__init__.py", line 10, in <module>
    from brainstorm.handlers.pycuda_handler import PyCudaHandler
  File "/home/arkade/Dropbox/codes/brainstorm/brainstorm/handlers/pycuda_handler.py", line 40, in <module>
    class PyCudaHandler(Handler):
  File "/home/arkade/Dropbox/codes/brainstorm/brainstorm/handlers/pycuda_handler.py", line 79, in PyCudaHandler
    array_type = pycuda.gpuarray.GPUArray
NameError: name 'pycuda' is not defined

Changing L79 in pycuda_handler.py to array_type = gpuarray.GPUArray leads to the bigger problem:

Traceback (most recent call last):
  File "/home/arkade/venv/py2/local/lib/python2.7/site-packages/_pytest/config.py", line 543, in importconftest
    mod = conftestpath.pyimport()
  File "/home/arkade/venv/py2/local/lib/python2.7/site-packages/py/_path/local.py", line 650, in pyimport
    __import__(modname)
  File "/home/arkade/Dropbox/codes/brainstorm/test/conftest.py", line 7, in <module>
    from brainstorm.structure.architecture import (
  File "/home/arkade/Dropbox/codes/brainstorm/brainstorm/__init__.py", line 5, in <module>
    from brainstorm.structure import *
  File "/home/arkade/Dropbox/codes/brainstorm/brainstorm/structure/__init__.py", line 4, in <module>
    from brainstorm.structure.network import Network
  File "/home/arkade/Dropbox/codes/brainstorm/brainstorm/structure/network.py", line 11, in <module>
    from brainstorm.structure.buffers import BufferManager
  File "/home/arkade/Dropbox/codes/brainstorm/brainstorm/structure/buffers.py", line 6, in <module>
    from brainstorm.handlers import default_handler
  File "/home/arkade/Dropbox/codes/brainstorm/brainstorm/handlers/__init__.py", line 10, in <module>
    from brainstorm.handlers.pycuda_handler import PyCudaHandler
  File "/home/arkade/Dropbox/codes/brainstorm/brainstorm/handlers/pycuda_handler.py", line 627, in <module>
    _mod = SourceModule(__softmax_kernel_code)
  File "/home/arkade/venv/py2/local/lib/python2.7/site-packages/pycuda/compiler.py", line 262, in __init__
    self.module = module_from_buffer(cubin)
LogicError: cuModuleLoadDataEx failed: invalid device context -

I wanted to continue the discussion about merging the forward- and backward buffers into a single buffer. The main advantage I see in this is that the code will be easier to read: whenever I show people brainstorm code, one of the first questions that pops up is "what do these buffers mean/why are there 2 of them" -- admittedly my sample size so far is 2 (I myself was also confused, so let's make it 3 ;) ). Also a pattern that emerges often within the code is something along the lines of:

    Ha = forward_buffers.internals.Ha
    dHa = backward_buffers.internals.Ha

Which indicates, to me at least, that the true name of backward_buffers.internals.Ha is should actually be buffers.internals.dHa, anyways.

If we implement this change, there are two ways to go:

1. Explicitly list the 'backward buffers' in the get_internal_structure:

   def get_internal_structure(self):
       internals = OrderedDict()
       internals['Ha'] = ShapeTemplate('T', 'B', self.size)
       internals['dHa'] = ShapeTemplate('T', 'B', self.size)
       return internals
   

2. Have a flag for each buffer requested via get_internal_structure, which indicates whether we need a "backward" buffer as well. In that case, just append a "d" to the name of the "forward buffer" we created. e.g.:

   def get_internal_structure(self):
       internals = OrderedDict()
       internals['Ha'] = ShapeTemplate('T', 'B', self.size, needs_gradient_buffer=True)
       return internals
   

The advantage of the 2nd approach is that it would still be fairly easy to lazily allocate backward buffers, and that it's less wordy. The downsize is that it might be a bit "magical". Which is why I actually prefer the first approach, even though it wastes memory and requires more typing. Typically, if you have enough memory the run a forward path, you'll have enough for the backward path as well, IMO.

Anyways... thoughts?

Issue A

Proposals:

1. Rename MonitorAccuracy to MonitorClassificationAccuracy. Additionally, later implement MonitorMeanSquaredError etc. We already have a separate MonitorHammingScore.
2. Implement MonitorScore/MonitorError (a better name) which allows the options to monitor various types of scores/errors such as classification error, hamming score, mean squared error etc.

2 will prevent code duplication. 1 might be more intuitive to use?

Issue B

Additionally, me and @Qwlouse talked some time ago about perhaps renaming Monitors (perhaps call them 'Hooks'?) This is because Monitors can do more than just monitoring stuff.

This hook stops the training when a certain accuracy, for example 97 %, is reached. Since it might be useful for others, I thought I'd share it.

Example usage:

trainer.add_hook(bs.hooks.StopAfterThresholdReached('validation.Accuracy',
                                                    threshold=0.97,
                                                    criterion='at_least'))

This needs a test and some documentation. If there would be documentation and tests for the existing EarlyStopper, then I could write similar docs and tests for this hook.

Would it make sense to provide basic plotting functionality to monitors that output per-epoch results (i.e. the accuracy monitor)?

• In one glance you would be able to obtain information about (speed of) convergence and performance.
• Allows easier sharing of results from one researcher to another.
• Would require matplotlib to be in place if the user chooses to make use of this functionality.

I am unsure if I am just missing something, but I could not find a simple method for doing a forward pass on some data and then extracting the results back to python. This is a function that does this, though I am not sure that I am entirely happy with how it returns data (as a list of lists of outputs of batches).

(If I did indeed miss the easy way to do this, my suggestion would be that one of the examples should do just that - i.e. say decode some MNIST test data and display prediction vs reference)

It looks like regression is looking for inputs names inputs_1 and inputs_2 instead of target and default.

• Is there a reason for this?
• Can you provide a simple regression example?

Also, I can't seem to get this to work even with the named changes. I get an error as such:

brainstorm.utils.LayerValidationError: Invalid in_shapes. SquaredDifference has no input(s) named "set([u'default', u'targets'])". Choices are: set([u'inputs_2', u'inputs_1'])

Sizing is: Input shape: (1411889, 1, 1) | Target shape: (1411889, 1, 1)

batch_size = 1
x_tr = x_tr[:, np.newaxis]
y_tr = np.roll(x_tr, -1, axis=0)

print("Input shape: %s | Target shape: %s" % (x_tr.shape, y_tr.shape))
getter_tr = Minibatches(batch_size, shuffle=False)

# ----------------------------- Set up Network ------------------------------ #

network = bs.tools.create_net_from_spec(task_type='regression',
                                        in_shape=batch_size,
                                        out_shape=batch_size,
                                        spec='L1 L512 D0.5 L256 D0.5 L512 L1') #also tried F1 for last layer
#                                        data_name='inputs_1',
#                                        targets_name='inputs_2')
network.set_handler(PyCudaHandler())
network.initialize(bs.initializers.Gaussian(0.01))

# ----------------------------- Set up Trainer ------------------------------ #

trainer = bs.Trainer(bs.training.MomentumStepper(learning_rate=0.01,
                                                 momentum=0.9),
                     verbose=True)
trainer.add_hook(bs.hooks.ProgressBar())
#out_name='Output_Conv.outputs.default')]
scorers = [bs.scorers.MeanSquaredError()] #targets_name='inputs_2'

We need a layer that has multiple inputs and just concatenates them along the last feature dimension. For CPU that one can be omitted, because the NumpyHandler supports slicing the features, but for usage with the PyCudaHandler this is the only way of merging the outputs of two layers.

With a fresh brainstorm installation and nothing in the DB, calling the Report Viewer leads to the following error:

** Error: Line 1062: Struct contents reference from a non-struct array object. ** ** Call stack: ** >bst_report.m>GetReport at 1062 ** >bst_report.m>Open at 630 ** >bst_report.m at 58 ** >bst_call.m at 28 ** >gui_brainstorm.m>@(h,ev)bst_call(@bst_report,'Open','current') at 124 **

After that a first protocol is created, it works.

https://github.com/IDSIA/brainstorm/blob/master/docs/installation.rst

Says "Instructions coming soon." for OS X. Any idea when "soon" will be? I'd love to play with brainstorm.

Thanks, Eric

The attribute call to gpudata is done by pycuda inside the function, no need to do it outside the function.

Before this change, using a merge layer with pycuda 2016.1 was giving the following errors in forward and backward pass:

  File "test.py", line 214, in main
    trainer.train(network, getter_tr, valid_getter=getter_va)
  File "brainstorm/training/trainer.py", line 99, in train
    self.stepper.run()
  File "brainstorm/training/steppers.py", line 103, in run
    self.net.forward_pass(training_pass=True)
  File "brainstorm/structure/network.py", line 430, in forward_pass
    layer.forward_pass(self.buffer[layer_name], training_pass)
  File "brainstorm/layers/merge_layer.py", line 52, in forward_pass
    buffers.outputs.default)
  File "brainstorm/handlers/pycuda_handler.py", line 323, in merge_tt
    block=block, grid=grid)
  File "/lib/python2.7/site-packages/pycuda/driver.py", line 383, in function_call
    handlers, arg_buf = _build_arg_buf(args)
  File "/lib/python2.7/site-packages/pycuda/driver.py", line 158, in _build_arg_buf
    raise TypeError("invalid type on parameter #%d (0-based)" % i)
TypeError: invalid type on parameter #0 (0-based)


  File "test.py", line 214, in main
    trainer.train(network, getter_tr, valid_getter=getter_va)
  File "brainstorm/training/trainer.py", line 99, in train
    self.stepper.run()
  File "brainstorm/training/steppers.py", line 104, in run
    self.net.backward_pass()
  File "brainstorm/structure/network.py", line 444, in backward_pass
    layer.backward_pass(self.buffer[layer_name])
  File "brainstorm/layers/merge_layer.py", line 59, in backward_pass
    buffers.input_deltas.inputs_2)
  File "brainstorm/handlers/pycuda_handler.py", line 364, in split_add_tt
    block=block, grid=grid)
  File "/lib/python2.7/site-packages/pycuda/driver.py", line 383, in function_call
    handlers, arg_buf = _build_arg_buf(args)
  File "/lib/python2.7/site-packages/pycuda/driver.py", line 158, in _build_arg_buf
    raise TypeError("invalid type on parameter #%d (0-based)" % i)
TypeError: invalid type on parameter #0 (0-based)

create_net_from_spec causes a parsing error depending on the presence or absence of whitespace, which is not according to the documentation ("Whitespace is allowed everywhere and will be completely ignored.").

Example script to reproduce the error:

import brainstorm as bs
print('This works')
net = bs.tools.create_net_from_spec('classification',4,5,'Ft200 D Ft200 D Ft200')
print('This does not work')
net = bs.tools.create_net_from_spec('classification',4,5,'Ft200DFt200DFt200')
print('Never reached')

Output:

This works This does not work Traceback (most recent call last): File "DoSomething.py", line 15, in net = bs.tools.create_net_from_spec('classification',4,5,'Ft200DFt200DFt200') File "build/bdist.linux-x86_64/egg/brainstorm/tools.py", line 614, in create_net_from_spec File "build/bdist.linux-x86_64/egg/brainstorm/tools.py", line 498, in create_layer KeyError: 'DF'

The LSTM backpass with Cuda does not work properly in the (rare, but possible) case that the length of an input sequence is 1. On the CPU, everything works fine.

The underlying reason is that an array of size (0,whatever) is allocated, which apparently works with Numpy, but results in an uninitialized array (gpudata==None) with PyCuda, on which subsequent operations ("fill" in the attached backtrace) fail. (To the best of my knowledge, this occurs first in lstm_layer.py, line 264 - flat_cell may have zero size. But I cannot guarantee that's the only occurrence.)

Further information: $ git log --oneline -n 1 a68bf03 Release 0.5 $ uname -a Linux nikola 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt7-1 (2015-03-01) x86_64 GNU/Linux

... and a backtrace, as well as a script to create the behavior (UseGPU and MakeItCrash must be set to 1!!) LSTMCrashWithGPUandLength1Seq.py.txt LSTMCrashBacktrace.txt

Bugfix: brainstorm.tools.extract crashed when a mask was provided by the data iterator.

I also reimplemented extract_and_save in terms of extract (fixes the very same bug). But this glosses over some of the original settings in extract_and_save (e.g. chunked storage), which I believe are unimportant. Still, you may want to have a look.

I also added a test (test_tools.py) for the bug and its fix.