Unsupervised Learning of Video Representations using LSTMs

python lstm_combo.py models/lstm_combo_1layer_mnist.pbtxt datasets/bouncing_mnist.pbtxt datasets/bouncing_mnist_valid.pbtxt 0 Using board 0 invalid device function Traceback (most recent call last): File "lstm_combo.py", line 405, in cm.CUDAMatrix.init_random(42) File "/home/weihaoxie/unsupervised-videos/cudamat/cudamat.py", line 382, in init_random raise generate_exception(err_code) cudamat.cudamat.CUDAMatException: CUDA error: no error

Hi,

I tried to run lstm_combo.py, but I got the error: OSError: dlopen(/Users/yantian/Google Drive/Yantian/DL/unsupervised-videos-master/cudamat/libcudamat.so, 6): image not found

Where can I get libcudamat.so? Thank you in advance!

Hi Emansim,

I try to train a model on MNIST dataset, but I cannot find libcudamat_conv_gemm.so.

Can you please help me to figure out what's happening?

Cheers,

Chengcheng

python lstm_combo.py models/lstm_combo_1layer_mnist.pbtxt datasets/bouncing_mnist.pbtxt datasets/bouncing_mnist_valid.pbtxt 0 Traceback (most recent call last): File "lstm_combo.py", line 1, in from data_handler import * File "/home/smile/project/unsupervised-videos-master/data_handler.py", line 3, in from util import * File "/home/smile/project/unsupervised-videos-master/util.py", line 4, in from cudamat import cudamat_conv_gemm as cc File "/home/smile/project/unsupervised-videos-master/cudamat/cudamat_conv_gemm.py", line 4, in _ConvNet = ct.cdll.LoadLibrary('libcudamat_conv_gemm.so') File "/home/smile/anaconda2/lib/python2.7/ctypes/init.py", line 443, in LoadLibrary return self._dlltype(name) File "/home/smile/anaconda2/lib/python2.7/ctypes/init.py", line 365, in init self._handle = _dlopen(self._name, mode) OSError: libcudamat_conv_gemm.so: cannot open shared object file: No such file or directory

Instead of relying on math libraries, it looks like you implement your own kernel for forward propagation. I heard that a small number adding with a large number can cause numerical problems. In your program, how do you handle this problem?

Hi, I have a problem with running the code. when I tried : python lstm_combo.py models/lstm_combo_1layer_mnist.pbtxt datasets/bouncing_mnist.pbtxt datasets/bouncing_mnist_valid.pbtxt 0 I got this error:

Using board 0 2048 2048 Traceback (most recent call last): File "lstm_combo.py", line 407, in main() File "lstm_combo.py", line 394, in main lstm_autoencoder = LSTMCombo(model) File "lstm_combo.py", line 21, in init self.lstm_stack_dec_.Add(lstm.LSTM(l)) File "/home/fahimeh/ProgramFiles/unsupervised-videos/lstm.py", line 21, in init self.w_dense_ = Param((4 * num_lstms, num_lstms), lstm_config.w_dense) File "/home/fahimeh/ProgramFiles/unsupervised-videos/util.py", line 29, in init self.dw_ = cm.empty_like(self.w_) File "/home/fahimeh/ProgramFiles/unsupervised-videos/cudamat/cudamat.py", line 1881, in empty_like cmat = empty(m.shape) File "/home/fahimeh/ProgramFiles/unsupervised-videos/cudamat/cudamat.py", line 1869, in empty raise generate_exception(err_code) cudamat.cudamat.CUDAMatException: CUBLAS error.

Could you please help me to fix it? Thanks a lot.

Hello Emansim and other friends,

I'm almost mad about this question and I've spent almost one week on it. This is a course project for one Master course in my university, and I have to reproduce the similar results to those in the paper.But this due time is drawing near while I'm stuck at the beginning of this project. I'm new to makefile, LSTM, google protobuf, so, please give me a hand.

Firstly, let me introduce my computer hardware and software configuration: Windows 7 Enterprise 64-bit operating system CUDA Toolkit v6.5 NVIDIA Quadro FX 4800 (Version 340.62) Microsoft Visual Studio Ultimate 2013 Microsoft SDK v8.1 (Framework v4.0) Python2.7.8 mingw32-make(after install mingw-w64-installer)

Secondly, let me introduce the steps to make Makefile in the folder cudamat:

1. open the cmd command shell;
2. cd to the directory of Makefile, i.e. C:\Users\david\Desktop\unsupervised-videos-master\cudamat
3. run the command: mingw32-make.exe -f Makefile The error below always occures:

Could anyone give me a hand? Thanks in advance.

@emansim

envy@ub1404:~/os_pri/github/unsupervised-videos$ python lstm_combo.py models/lstm_combo_1layer_mnist.pbtxt datasets/bouncing_mnist.pbtxt datasets/bouncing_mnist_valid.pbtxt 1 invalid device ordinal Traceback (most recent call last): File "lstm_combo.py", line 402, in board = LockGPU(board=board_id) File "/home/envy/os_pri/github/unsupervised-videos/util.py", line 144, in LockGPU cm.cuda_set_device(board) File "/home/envy/os_pri/github/unsupervised-videos/cudamat/cudamat.py", line 2273, in cuda_set_device raise generate_exception(err_code) cudamat.cudamat.CUDAMatException: CUDA error: no error envy@ub1404:~/os_pri/github/unsupervised-videos$

Dear developers,

I am a newbie in LSTM so I read your code carefully to gain a deep understanding.

Basically, the code for Fprop runs like: for each batch: for each LSTM: LSTM.Fprop() where batch is the "input_frame" with shape (batch_size, input_dims) and LSTM units are stored in list "models_".

Let's take video patch as an example, i.e., "python lstm_combo.py models/lstm_combo_1layer_ucf101_patches.pbtxt datasets/ucf101_patches.pbtxt datasets/ucf101_patches_valid.pbtxt 1".

If you add print statements to show their shape or length, it outputs: len(models_) = 1 input_frame= (100, 3072)

This means there is only one LSTM object in the "models_" list. Well, if you look at Fprop of LSTM class, it allocates memory for whole network, for example, "w_input_" variable.

My questions are:

1. Is each LSTM object a stacked LSTM network?
2. According to your LSTM Fprop implementation and your cuda kernel kLSTMFprop(), are you multiplying different "w" with "x" first and then in cuda kernel summing different components up?
3. Just to make sure, "ucf101_sample_train_patches.npy" has shape (9000, 20, 3, 32, 32) where 20 means 20 frames right? What is following code doing?

for t in xrange(self.enc_seq_length_): self.lstm_stack_enc_.Fprop(input_frame=self.v_.col_slice(t * self.num_dims_, (t+1) * self.num_dims_))

Any help will be appreciated. Thanks

bounce-off problem

digit moving is in fact based on digit patch(2828), not digit itself. So the bounce-off is exactly based on digit patch. In most case, digit will bounce back before digit arrives at the edge of outer box(6464) ,due to white padding in digit patch. Does it matter for the training? What problem will it cause?

digit overlap problem

data_handler.py about line 417

    """ Put b on top of a."""
   def Overlap(self, a, b):
    # S1: original by author, can not make sure that b is on top of a
    #return np.maximum(a, b)

    #S2: digit patch overlap,consistent  with bounce-off process 
    #return b

    # S3: digit overlap, but inconsistent with bounce-off process
    return np.where(b==0,a,b)

As mentioned by the notes above,

• S3 is truly digit overlap
• S2 is digit patch overlap
• S1 is something else, but it is the original code in the project

Is that a problem?

envy@ub1404:~/os_pri/github/unsupervised-videos$ python lstm_combo.py models/lstm_combo_1layer_mnist.pbtxt datasets/bouncing_mnist.pbtxt datasets/bouncing_mnist_valid.pbtxt 1 Traceback (most recent call last): File "lstm_combo.py", line 1, in from data_handler import * File "/home/envy/os_pri/github/unsupervised-videos/data_handler.py", line 3, in from util import * File "/home/envy/os_pri/github/unsupervised-videos/util.py", line 16, in import config_pb2 ImportError: No module named config_pb2 envy@ub1404:~/os_pri/github/unsupervised-videos$

I am using an Amazon Machine Image (AMI) to launch a GPU instance on Amazon EC2 with CUDA 7.5 and the required libraries installed. When I type make in your cudamat directory, everything appears to work. I am able to compile the .proto file and download the datasets. However, when I attempt to train a model, I get this error:

ubuntu@ip-172-31-9-162:~/unsupervised-videos-master$ python lstm_combo.py models/lstm_combo_1layer_mnist.pbtxt datasets/bouncing_mnist.pbtxt datasets/bouncing_mnist_valid.pbtxt 1 invalid device ordinal Traceback (most recent call last): File "lstm_combo.py", line 402, in board = LockGPU(board=board_id) File "/home/ubuntu/unsupervised-videos-master/util.py", line 144, in LockGPU cm.cuda_set_device(board) File "/home/ubuntu/unsupervised-videos-master/cudamat/cudamat.py", line 2273, in cuda_set_device raise generate_exception(err_code) cudamat.cudamat.CUDAMatException: CUDA error: no error

Is your cudamat setup compatible with CUDA 7.5? Is it necessary to use your custom directory?

Can you help me find the source of this error? I'd appreciate it very much.

Best, Tina

Hi, Thank you very much for providing the code for the paper. I have a small doubt regarding using your code to predict video frames on some other video dataset. Could you kindly share some insight on how to do that? Also, could you kindly tell me if the .npy file contains videos (in frames) of the entire videos in dataset like in a 5-D matrix (video_number,frame_no,channels,height,width) format? Thank you very much.

Instead of giving videos, how do we test with normal matrices? for examples if i am giving 5x5 matrix then is there a way to get (let's say) a 8x8 matrix? (trained with other 5x5 matrix)

Hello, When I try to implement

python lstm_combo.py models/lstm_combo_1layer_mnist.pbtxt datasets/bouncing_mnist.pbtxt datasets/bouncing_mnist_valid.pbtxt 1 I get following error: OSError: /nas/ei/home/ga85pav/unsupervised-videos/cudamat/libcudamat.so: cannot open shared object file: No such file or directory

I don't see the libcudamat.so in cudamat folder but it is mentioned in Makefile about it. Am I missing something?

Could someone help me on it.

2. Also, is it possible to skip training, Are there pre-trained models available anywhere for UCF dataset or MNIST dataset?

Thanks!!

when I run python lstm_combo.py models/lstm_combo_1layer_mnist.pbtxt datasets/bouncing_mnist.pbtxt datasets/bouncing_mnist_valid.pbtxt 1 I got this errror "no kernel image is available for execution on the device" my environment is ubuntu16.10/1080ti/cuda9.1/anaconda2/python2.7

1. The authors mentioned about "We initialize an LSTM classifier with the weights learned by the encoder LSTM from this model. "in their paper, but I am a beginner and I don't understand how to initialize the lstm classifier?

2. The Figure 6 shows that the lstm classifier is two layers ,could your tell how to implement it ?

Is there anyone can help me with this? Thanks a lot.

@emansim