In response to: https://github.com/lisa-lab/pylearn2/issues/1004

When returning data_specs, DefaultDatasetMixin and get_monitoring_data_specs now call get_target_space instead of get_output_space. The base model class's get_target_space defaults to get_output_space unless the model has specified the _target_space attribute.

Softmax now has 2 new inputs: binary_targets and target_dim which allow the user to have a softmax with IndexSpace targets. This saves a lot of computation time not transfering and multiplying one-hot vectors.

• [x] Introduce a SequenceSpace class which can represent a sequence of elements in any other space by prepending a time-dimension.
• [x] Monkey patch MLP to deal with SequenceSpace input by reshaping (time, batch, data) to (time * batch, data) and changing spaces for non-RNN-friendly layers.
• [x] RNN-friendly subclasses of SequentialSubsetIterator and ShuffledSequentialSubsetIterator which can create batches of samples of the same length, ensuring that no padding is required.
• [x] ~~RNN-friendly subclasses of e.g. ForcedEvenIterator, SubsetIterator, etc.~~
• [x] ~~Find a cleaner way for the correct iterator to be called. Right now each dataset needs to construct the subset iterator explicitly and pass sequence_lengths, which would have to be done for each sequence-dataset. This might not be needed if we implement zero-padding and mask instead (see below).~~
• [x] Patch FiniteDatasetIterator to shuffle the dimensions of SequenceSpace data from (batch, time, data) (the way the dataset stores it) to (time, batch, data) (the way the model consumes it)
• [x] ~~Implement zero-padding and a mask (see above)~~
• [x] Gradient clipping as a cost wrapper
• [ ] Port a whole bunch of things from GroundHog and @jych's code e.g. ~~gradient clipping (as a ModelExtension?)~~, LSTM, etc.
• [x] Costs
• [x] Cost-matrix wrapping

The implementation is based heavily on the AdaDelta learning rule. Tests still needed -- should hopefully be able to get around to that over the course of the next day.

The motivations and math behind RMSProp are described by Geoffrey Hinton at the lecture slides below: http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf.

This is a clean subset of the variational autoencoder framework I've been working on lately. I'm opening the pull request now to get feedback, even though there are no tests written yet.

In order to keep the code modular, I'm re-using the MLP framework for both the encoder and the decoder networks, which has proven to be quite useful in my experiments.

One design decision I made (and for which I'd like to know your thoughts) is using multiple inheritance. The BaseVAE base class defines an internal interface, which I broke down in two distinct parts that are handled by separate subclasses of the base class:

1. Input-related methods, such as sampling from the conditional p(x | z) and computing log p(x | z), are handled by subclasses in the visible module.
2. Latent-variable-related methods, such as sampling from the posterior q(z | x) and the prior p(z), computing the KL-divergence between posterior and the prior, and computing their respective log-probabilities, are handled by the latent module.

A helper method allows dynamical inheritance, so that the user can mix-and-match conditional distributions p(x | z) and posterior/prior distribution on latent variables (e.g. having real-valued inputs vs. binary inputs while keeping the same gaussian prior/posterior).

https://github.com/lisa-lab/pylearn2/issues/1068 it shows the list of possible monitoring channels and, if the user doesn't specify anything, pylearn2 works as it works now, otherwise it prints/calculate only the log_monitors passed. It solves also the problem of the slowness of ipython.

Example:

!obj:pylearn2.train.Train {
dataset: &train !obj:pylearn2.datasets.csv_dataset.CSVDataset {
    path: 'train.csv'
},
model: !obj:pylearn2.models.mlp.MLP {
    layers: [
             !obj:pylearn2.models.mlp.Sigmoid {
                 layer_name: 'h0',
                 dim: 300,
                 sparse_init: 15
             },
             !obj:pylearn2.models.mlp.Linear {
                 layer_name: 'y',
                 irange: 0.5,
                 dim: 1
             }
            ],
    nvis: 20,
},
algorithm: !obj:pylearn2.training_algorithms.sgd.SGD {
    batch_size: 10,
    learning_rate: .001,
    monitoring_dataset:
        {
            'train' : *train,
            'test'  : !obj:pylearn2.datasets.csv_dataset.CSVDataset {
                            path: 'testY.csv'
                        },
        },
    monitoring_channels: ['train_objective'],
    termination_criterion: !obj:pylearn2.termination_criteria.EpochCounter {
        max_epochs: 940
    },

},
extensions: [
    !obj:pylearn2.train_extensions.best_params.MonitorBasedSaveBest {
        channel_name: "test_objective",
        save_path: "best.pkl"
    }
],
save_path: "softmax_regression.pkl",
save_freq: 1

}

d:\kit\pthreads-win32-VC-x64\ is where the pthreadVC2.lib, pthreadVC2.dll and pthread.h reside on my system. User should be able to specify the header file location, library name and library location via config attributes or environment variable (or better yet, code should be made portable, perhaps by using C++11 standard threading classes).

Also, defines that trigger symbol exporting should be passed as -D args to compiler.

SUMMARY OF THE MAILING LIST ABOUT SOFTWARE VERSION:

Long story short, I switched computers and toolchain versions and got a very strange behaviour: something % 1 = something, instead of something % 1 = 0, which caused ops to fail. After updating msvc from 2008 to 2010 and cuda from 4.2 to 5.0 everything went fine. I also noticed another slightly confusing issue: the console output didn't get updated regularly and training seemed to remain stuck after a few epochs.

The code in Theano was the same. It just had a couple of extra printf statements to help debugging.

I don't really know what caused the code to produce such a strange output, and I couldn't reproduce it, despite trying to compile the same piece of code (including variable types) using the same compiler arguments. This incident kind of made me lose a tiny bit of faith in the deterministic nature of the compilation process. It might have been some misconfiguration of the toolchain (the machine is used for a number of other projects using CUDA as well, and I'm not familiar with its environment configuration), or a bug in nvcc4 or msvc2008, or some temporary files I might have accidentally copied along when switching machines.

When I exchange the "MaxoutConvC01B" with"MaxoutLocalC01B Im assuming that my yaml files still will work. I do however get the following problems.

1. MaxoutLocalC01B assumes a "max_filter_norm" argument while the MaxoutConvC01B class wants a "max_kernel_norm" argument

Changing max_kernel_norm into max_filter_norm , lets me go a bit forward but I then get got an unexpected keyword argument 'kernel_stride' in : /linear/local_c01b.py", line 54, in init

Seems like local_c01b.py asuumes localDot.init can take kernel_stride as argument but this isnt implemented in Local dot

-when called from local_c01b.py LocalDot.init(self, filters=filters, irows=image_shape[0], icols=image_shape[1], kernel_stride=kernel_stride, padding_start=pad, message='')

-definition in localdot.py def init(self, filters, irows, icols=None, subsample=(1, 1), padding_start=None, filters_shape=None, message="")

This PR includes all of the code for working (unsupervised) GSN's.

The main contributions are: pylearn2.models.gsn.GSN : This is the actual model class. It is heavily reliant on pylearn2.models.autoencoder.Autoencoder.

pylearn2.costs.gsn.GSNCost : cost class for the model

pylearn2.models.tests.test_gsn : driver for GSN testing and sampling. Also serves as tests and example.

pylearn2.distributions.parzen : code for Parzen estimator, which is useful to evaluate sampling algorithms. This code is pretty much directly copied from https://github.com/yaoli/DSN (the original implementation of GSNs)

functions in pylearn2.utils.image : Makes it easy to create images such as http://i.imgur.com/EvlEvBW.png . Again, this code was directly copied from the original GSN's implementation.

several new corruptors in pylearn2.corruption : Particularly, I added the SaltPepperCorruptor, BinomialSampler, MultinomialSampler, and ComposedCorruptor (which just composes corruptors).

NOTE: I did slightly change the corruptor interface (modifying what a private method does). Previously, _corrupt had to handle both single tensor objects or list of tensor objects (by mapping over them). This was a bit ugly because every Corruptor subclass included this logic in _corrupt. I changed this so that _corrupt only operates on a single tensor, and call (defined only on the Corruptor base class) handles the mapping of lists. This could theoretically break something, but from ack'ing around the codebase I didn't see anything this would break. Does pylearn2 have a test suite that should be run before merging this in? As far as I know, this is the only change that could break things.

I wrapped sklearn.metrics.roc_auc_score as a Theano Op and wrote a TrainExtension class that calculates ROC AUC on each monitoring dataset. It's recommended to use monitoring_batches=1 with this extension, since ROC AUC cannot be calculated unless both classes are represented in a batch (but it fails gracefully, returning nan for that channel value).

See test_roc_auc.py for a simple example.

To resolve issue #1053 by @cancan101.

Currently, this appears in numerous places: np.any(np.isnan(X)). This requires creating a large array of booleans matching the shape of X. More efficient would be: np.isnan(np.min(X)).

See: http://stackoverflow.com/questions/6736590/fast-check-for-nan-in-numpy

This PR adds:

• The capability to perform cross-validation using training, validation, and test sets.
• Support for TrainCV extensions, including a TrainCV version of MonitorBasedSaveBest that saves the best model from each CV fold after all training is complete.

The cross-validation iterators in sklearn only return train/test splits. The new subset iterators (and their corresponding dataset iterators) in this PR return train/valid/test splits by starting with a standard train/test split and further dividing the train subset into a train/valid split.

Additionally, the cross-validation tests have been reorganized into several files.

I avoided all issues with theano.compat.six versus six with these changes. I left a comment with all the details on what I did in the issue related to this. I'll attach it here when I find it. Just ran a couple of bash scripts.

import pylearn2.config.yaml_parse test_str ='!!python/object/apply:os.system ["ls"]' test_load = pylearn2.config.yaml_parse.load(test_str)

Hi, there is a vulnerability in load methods in pylearn2.config.yaml_parse.py,please see PoC above. It can execute arbitrary python commands resulting in command execution.

When I want to import pylearn2, erroe "ModuleNotFoundError: No module named 'theano.compat.six'" always come. It's a error about version not matching. Of course, you can change "from theano.compat.six.moves import input" to "from six.moves import input" . But source code maybe contained many mismatching statements, It's hard to change all of them.

According to this issue, import statements with theano.compat.six should be replaced by import six (or from six import ...). Replacing all import statements appears to have solved the issue.

I receive the following TypeError when I try to import CIFAR10 data via the pylearn2 API:

Traceback (most recent call last): File "my_binary_net.py", line 27, in train_set = CIFAR10(which_set="train", start=0, stop=train_set_size) File "/Users/Jackson/pylearn2/pylearn2/datasets/cifar10.py", line 85, in init x = numpy.zeros((lenx, self.img_size), dtype=dtype) TypeError: 'numpy.float64' object cannot be interpreted as an integer

I fixed the error by typecasting lenx, which is originally np.float64, to int: lenx = int(numpy.ceil((ntrain + nvalid) / 10000.) * 10000)

Hi All,

I am using trying to run BinaryNet which uses PyLearn2. When I am running the mnist.py of BinaryNet, it is giving me PyLeran2 related error as follows. I am on Windows 7 machine and I am the admin of this machine. I am wondering if anyone has suggestions ?

Thanks in advance for your help.

Using gpu device 0: Quadro K620 (CNMeM is disabled, cuDNN not available) C:\Users\jmatai\AppData\Local\Continuum\Anaconda2\lib\site-packages\theano\tensor\signal\downsample.py:6: UserWarning: downsample module has been moved to the t heano.tensor.signal.pool module. "downsample module has been moved to the theano.tensor.signal.pool module.") batch_size = 100 alpha = 0.1 epsilon = 0.0001 num_units = 4096 n_hidden_layers = 3 num_epochs = 1000 dropout_in = 0.2 dropout_hidden = 0.5 activation = binary_net.binary_tanh_unit binary = True stochastic = False H = 1.0 W_LR_scale = Glorot LR_start = 0.003 LR_fin = 3e-07 LR_decay = 0.990831944893 save_path = mnist_parameters.npz shuffle_parts = 1 Loading MNIST dataset... Traceback (most recent call last): File "mnist.py", line 93, in train_set = MNIST(which_set= 'train', start=0, stop = 50000, center = False) File "c:\projects\python_packages\pylearn2\pylearn2\datasets\mnist.py", line 93, in init topo_view = read_mnist_images(im_path, dtype='float32') File "c:\projects\python_packages\pylearn2\pylearn2\utils\mnist_ubyte.py", line 94, in read_mnist_images with open_if_filename(fn, 'rb') as f: File "c:\projects\python_packages\pylearn2\pylearn2\utils\mnist_ubyte.py", line 46, in enter self._handle = open(self._f, self._mode, self._buffering) IOError: [Errno 13] Permission denied: 'C:\projects\repos\dl\data/mnist/train-images-idx3-ubyte'