Hey, I am trying to integrate your convLSTM cell into an existing model I have. I did this in the following way:

    def forward(self,x): 
        out = my_model(x)
        out = out.unsqueeze(1) #make sure dimensions fit
        out, self.hidden = self.convlstm(out, self.hidden)
        return out[-1].squeeze(1)

self.hidden is none in the first run, but not none in the second, leading to:

    # Implement stateful ConvLSTM
    if hidden_state is not None:
        raise NotImplementedError()

which is in your ConvLSTM module (line 141, 142)

would this be implemented just by:

    # Implement stateful ConvLSTM
    if hidden_state is not None:
        hidden_state=hidden_state`

or am I misunderstanding something? What is supposed to happen here?

Thanks for any help :-D

I have a question regarding your implementation: As I understood the original convolutional lstm formulation is as follows:

But in your implementation, u used only one convolution layer. I don't understand how these 2 correspond with each other. because in the formulation, c is only used in the Hadamard product and not in convolutions, but here c and h are both used in convolutions. in fact, all weights are shared for all 4 formulas, although there are 11 weights in the original formula.

This code snippet taken and changed slightly from the docstring fails for me:

import torch
from CONVLSTM_Implementation import ConvLSTM
x = torch.rand((32, 10, 64, 128, 128))
convlstm = ConvLSTM(64, 16, (2, 2), 1, True, True, False)
_, last_states = convlstm(x)

with the following Error:

c_next = f * c_cur + i * g
RuntimeError: The size of tensor a (129) must match the size of tensor b (128) at non-singleton dimension 3

But I don't really know why. I guess I will use a different kernel_size for now.

First off thank you for the implementation. Are Hadamard products in the paper missing? Please see the image below and paper that I am referring to for clarification.

Image for equations Paper

Hey I was not sure how the logic in line 49 works

cc_i, cc_f, cc_o, cc_g = torch.split(combined_conv, self.hidden_dim, dim=1)

Since each of the 4 gates has operations of weights with inputs, how is the order of split determined? Why not something like cc_g, cc_f, cc_i, cc_go = torch.split(combined_conv, self.hidden_dim, dim=1)? I am a bit confused how the LSTM equations in https://pytorch.org/docs/stable/nn.html#torch.nn.LSTMCell are implemented here.

Thanks in advance.

Sorry, my English is not very good. Why split_size_or_sections is self.hidden_dim? There are only 4 variables to receive the result, I think self.hidden_dim should be changed to 4

44 cc_i, cc_f, cc_o, cc_g = torch.split(combined_conv, self.hidden_dim, dim=1)

Mods:

• Removed Variable calls (old pytorch version)
• Removed dependency on image size (now inferred by the batch size)
• Clarified output and docstring
• Fixed device hardcoded for hidden states
• Some PEP8 stuff

If you run the example toy data in the script you will get the above error in pytorch 0.40. I am not sure if the eailer version will cause this issue. How can you make sure this is working?

I saw that in the code, hidden_state is not implemented:

    def forward(self, input_tensor, hidden_state=None):
        """

        Parameters
        ----------
        input_tensor: todo
            5-D Tensor either of shape (t, b, c, h, w) or (b, t, c, h, w)
        hidden_state: todo
            None. todo implement stateful

meanwhile, hidden_dim is given. What is the difference between those two variables?

model = ConvLSTM(input_dim=channels, hidden_dim=[64, 64, 128], kernel_size=(3, 3), num_layers=3, batch_first=True[missing comma here] bias=True, return_all_layers=False)

sorry, i didn't really understand conLSTM when i use keras layers ConvLSTM2D(filters = 128, kernel_size=(3, 3), padding='same', return_sequences = False, go_backwards = True,kernel_initializer = 'he_normal' ),such as input shape is (batch,2,h,w,channel),2 i guess is time,and the output is (batch,h,w,128) but in your code ,i didn't get same shape, can you help me to get the shape like keras convLSTM,thx

when i build my model with convlstm in the function def forward(self, x): summary[m_key]["input_shape"] = list(input[0].size()) in torchsummary.py the input is empty tuple, so the program error with '{IndexError}tuple index out of range'