Hi, thank you for your fabulous work. I've been trying to implement a training code for the pytorch version of liteflownet.

Everything seems to be working fine except def foward() defined in Regularization class. (It was correlation module that I suspected the most but it is working fine.) I get "nan" loss value after one iteration. It seems that gradient become nan at the foward block.

If I change line 279 to tensorDifference = (tensorFirst - tensorSecond).pow(2.0).sum(1, True).sqrt() loss never becomes "nan".

I am kind of having hard time with the code because I've never used pytorch (I am getting it though). Do you have any idea about this? Thank you

Hello, in issue #11, Is there any reason why do scaling with ((tensorInput.size(3) - 1.0) / 2.0) instead of (tensorInput.size(3) - 1.0)? Does the line assume that the input tensorflow stay in [-(w-1)/2, (w-1)/2] so result will be [-1, 1]?

Thank you for critically examining my code! Please note that the estimated tensorFlow itself can have negative values and the only reason for the division is to scale tensorFlow in accordance with the image size. Try your suggested version, execute the provided run.py, and examine the result, it will probably have little to no meaning anymore.

I used the visualize method to convert flo to png. For pytorch-pwcnet, the result is correct, however the visualization for pytorch-liteflownet is all black picture, there must be something wrong, could you check the result?

Dear Simon Niklaus, Thank you so much for sharing the testing code for liteflownet.

I tried to write the training code of liteflownet by modifying your testing code. I enablebed gradient computation torch.set_grad_enabled(True) and did other necessary changes, but found RuntimeError: CUDA error: out of memory at following line in "class Regularization" tensorDist = self.moduleDist(self.moduleMain(torch.cat([ tensorDifference, tensorFlow - tensorFlow.view(tensorFlow.size(0), 2, -1).mean(2, True).view(tensorFlow.size(0), 2, 1, 1), self.moduleFeat(tensorFeaturesFirst) ], 1)))

Can you guide, how can i fix this problem. Best, Farooq

Hi, I have questions regarding the data augmentation that normalize the input tensor value using the mean RGB.

I notice that you put the normalization (mean value subtraction) directly into the model class.

1. If we want to train the model, do we have to include this mean subtraction also? Or is it only for inference?

2. Suppose that I want to use your code to train a LiteFlowNet from scratch (with my own datasets*), does it means that I have to change the mean value according to my own datasets?

*the mean value of my datasets are quite lower than the one that used in the original LiteFlowNet model (e.g., 0.171..)

Thank you for your attention. Regards

Could you please explain what the correlation.py script does? I've seen it being used in Flownet2 and PWCNet as well. Can someone link a detailed explanation of the exact steps. I wish to write an equivalent code for CPU, and all current implementations require cuda. Thanks

Hi @sniklaus Thank you so much for your code! Sorry for the really sill question. I was able to set up your code and got it working for the sample images (1024x436) you've provided. I wish to estimate the flow for images that are (512 x 384). I commented the following lines from your code:

assert(intWidth == 1024)
assert(intHeight == 436)

And ran it, but I'm getting the following error:

Traceback (most recent call last):
  File "run.py", line 379, in <module>
    tenOutput = estimate(tenFirst, tenSecond)
  File "run.py", line 356, in estimate
    tenPreprocessedFirst = tenFirst.cuda().view(1, 3, intHeight, intWidth)

RuntimeError: shape '[1, 3, 384, 512]' is invalid for input of size 786432

Could you kindly advise me on how to fix this? Do we have to modify the architecture of the network, if we are only testing it out? Thank you, Abhinav

Hi, thanks for your great work. I run python run.py to test the images in floder./images, and the result is normal. When I test on Sintel clean, the result is good for some images, but some is very bad. I don't know why, could you please give me some suggestions? And what's your result AEPE on Sintel dataset? good result: （index and EPE are as follow） bad result: The AEPE for all:

Hi Simon, In parallel to translating Hui's pretrained Caffe model to Pytorch, did you try to train your LiteFlowNet implementation from scratch? I tried to train it (from scratch, end-to-end) on FlyingChairs using the Adam optimizer with different learning rates (lr=0.01, lr=0.001) but could not have the L2 loss decrease properly (neither on the trainset nor the val set). Thanks for any advice. Julien

First of all, thanks for your LiteFlowNet reimplementation in PyTorch. I wonder how do you convert the pre-trained model to PyTorch?

Since I'm trying to convert this model from Caffe to PyTorch also, could you please explain the procedure, or maybe refer to a certain method that you used? That would be very helpful.

Regards

For the Liteflownet, the resolution of the output of the optical flow when inferring with liteflownet is the half of the original image pair, and then the output is upsampled to the same size as the original image. However, why not directly output the optical flow whose size is the same as the original image? And I notice that PWC-Net also uses the similar manner. Thank you!