Do you have the kth_action dataset ? I use the kth_action dataset from the official website, and preprocess the video to frames. But I cannot get the training set of 108,717 and test set of 4,086 sequences as the paper mention. I get like below:

training set:
there are 183861 pictures
there are 30379 sequences
test set:
there are 105855 pictures
there are 17145 sequences

Can u give me the preprocess method or the preprocessed dataset ？

what the role of reshape_patch in core/utils/preprocess? I found that adding the patch_size parameter can significantly speed up the training process and reduce the cuda memory.

And I test function reshape_patch using the code following:

    img = cv2.imread('cat.jpeg', 0)
    img = img[np.newaxis, np.newaxis, :, :, np.newaxis]
    img_patched = reshape_patch(img, 3)

and I show the img(a gray image) and img_patched:

So is this function used to reduce the spatial resolution of the image?

And thanks for your good work!

@wuhaixu2016 Hello, thank you for sharing your work. My test results on the KTH dataset are shown below. My understanding is to input 20 frames of images and predict 19 frames. However, why are these two sequences almost identical instead of predicting future actions.

Looking forward to your reply, thank you so much!!!

Hi,

I ultilized predRNN and your traininig strategy (i.e., combine reverse schedule sampling and schedule sampling) to give a soil moisture forecasting. We ultilized 7 days soil moisture to predict it on future 7 days. However, I found the prediction images can't capture the evolution of soil moisture during forecasting steps, and give the same pattern of soil moisture on step 8 (see attached figure).

Can you give me some suggestions? Thanks a lot !

Lu

'Hi, this code repo is correct. We will rephrase the paper soon.

Originally posted by @wuhaixu2016 in https://github.com/thuml/predrnn-pytorch/issues/26#issuecomment-968199072'

Thanks for your reply. I couldn't reproduce the performance on Moving_MNIST in PredRNN++ anyway. So I would like to know whether the Moving_MNIST dataset volume in PredRNN++ is the same as the code repo? If not, which volume setting is the experiment in PredRNN++ based on? Looking forward to your reply. Thanks very much!

Hi there,

I am having issues using the action-conditional PredRNNV2 for inference.

The way it seems to work (action_injection=concat): Load the actions, grid-repeat them and concat the actual video data and the resulting action tensor channel-wise. Then, use reshape_patch() and pass the input to the model, resulting in a tensor of shape [batch, seq_length, height // patch_size, width // patch_size, (img_ch + action_ch) * patch_size ** 2].

For the action-conditional PredRNNV2 model however, the parameter num_action_ch is used directly for the input channels for the conv layers instead of num_action_ch * patch_size ** 2. For me, this leads to runtime shape mismatches in forward(). Is this an error or did I get it wrong somehow?

Hi, I'm using predrnn to predict future frames of experimental acquired data. To augment the data, I had cut each frame into overlapping tiles, and trained predrnn on those tiles.

Currently I'm testing how good predrnn predicts. To do so, I'm using the trained model on data, test.npz, that had not previously seen. This data is also composed of overlapping tiles, the collection of which represents different frames.

The problem I have is this: once predictions are made, the predicted tiles are generated in the corresponding folders, however these tiles are somewhat in a random order, or at least I'm sure they're not in the same order as in the test.npz file that was fed to the trained predrnn model. My question is, is it possible to preserve the same order? The reason for this is that I want to reconstruct frames from the predicted tiles, and if the tiles' order changes, it's very difficult to do a proper reconstruction.

Thanks Miguel

Hi, I see that there are only training and test datasets for above KTH_action, TaxiBJ, and Human3.6M. So, what's about the validation dataset or there is no validation sequences in these datasets? Thanks!

Hi, Thanks for the repo. I could get good results with your pretrained model. May I know how we can use the pretrained model for 3 digits Moving MNIST?

Hi, Thank you for sharing with the great work. I wonder if the length of input sequence for training and testing can be different? To be specific, for training, a length-10 video clip is fed into the network, while for testing, only one frame is used as input.

Hi, I tried testing the pretrained model of predrnn-v2 for moving-mnist dataset. But I get the following error. Can you please check the reason?

RuntimeError: Error(s) in loading state_dict for RNN: Unexpected key(s) in state_dict: "adapter.weight".

Thanks!!

"from skimage.measure import compare_ssim" is deprecated. Change it to: "from skimage.metrics import structural_similarity as compare_ssim"

When I try to execute the Moving MNIST script using a pretrained model (with arguments "--save_dir" and "--pretrained_model"), the error in the title appears. I believe the lines of code posted below cause the error in the title.

https://github.com/thuml/predrnn-pytorch/blob/36ba2b63fa96da3a60c32127e48f517e04062201/run.py#L210-L212

Hi, As mentioned in previous issue here, may I know how did you include items like "clips", "dims" in the MNIST dataset?

Thank you!

In your article you wrote in the traffic4cast section:

To cope with high-dimensional input frames, we apply the autoencoder architecture of U-Net [88] to the network backbone of PredRNN. Specifically, the decoder of U-Net contains four ST- LSTM layers, and the CNN encoder takes both traffic flow maps and spatiotemporal memory states as inputs.

The code for this model is not available on this GitHub repo, can you make it available please.

Thanks a lot!

Kind regards, Sébastien de Blois