Hi Clément,

I just noticed that you are working on refactoring the source code! Thanks a lot!

Btw, I am currently using a modified version of your previous implementation, and I found that a simple modification on the loss design produced better results.

In your loss implementation (multiscaleloss.py), it first calculates the L2 distance (EPE) between the GT and output, and then it averages them over the image. However, when just "summing up the EPE", the network performs roughly similar to the original implementation. (Actually I got near 2.2 EPE in the FlyingChairs from this modification + some additional minor things.)

I haven't thoroughly checked the original FlowNetS implementation in Caffe, but when looking at the scale of the loss function of theirs, I thought that summing up the L2 loss over the image seems the way that the original implementation took.

Could you check whether this is the case?

Thanks, Jun

Hello, i download the flownets_bn_EPE2.459.pth.tar and flownets_EPE1.951.pth.tar And when i decompress it, the file is broken. Can you share the two files again? Thanks very much.

Thank you for this simple and readable code. I am also glad that this works with python 3.5 pytorch 0.3. I look forward to you adding other networks such as FlowNetC and FlowNet2.0. Also other metrics such as percentage outliers would be a great addition.

KITTI flow GT is sparse, but this is not considered in flow reading or in training. I suggest the following changes..

In KITTI.py, the ground truth flow reading is not correct. By looking at the KITTI flow reading script and the readme.txt there, this is what I wrote.

def load_flow_from_png(png_path):

# read using cv2 and convert from bgr to rgb
# scipy cannot handle 16 bit images, hence cv2 is used.
flo_img = cv2.imread(png_path,-1)
flo_img = flo_img[:,:,::-1].astype(float)

# see the readme file in KITTI devkit and the flow reader functions
mask = np.minimum(flo_img[:,:,2],1)
not_valid = (mask == 0)
valid = (mask != 0)
flo_img = flo_img[:, :, 0:2]
flo_img = flo_img - 32768
flo_img = flo_img / float(64.0)

# value 0 is used to indicate invalid flow.
# flow that is actually valid and zero is set to a very small value
eps = 1e-10
flo_img[np.abs(flo_img) < eps] = eps

# invalid flow is indicated by 0
flo_img[not_valid, :] = float(0.)
return flo_img

Apart from the above function, the sparse flag has to be passed into several functions. I added a flag called sparse_gt

if args.sparse_gt is None: args.sparse_gt = ('KITTI' in args.dataset)

and this flag is passed to all the relevant functions such as: multiscaleEPE, one_scale, realELE, EPE etc.

With these changes, I am getting more meaningfull EPE values.

Kindly fix this issue.

Hi~ I have training my model with the same config as yours. But, the decay_loss decrease too faster, and after about 6000 iterations, the loss become nan. So I want know your training configurations, are they the default configurations in your main.py?

Hi @ClementPinard, i have a small question. Why do you normalize disparities / flows by std=20. Where did you find this in the original code? Thank you!

@ClementPinard Did you cross-checked losses during training in caffe and pytorch? Are they similar? In my case (DispNetCorr1) in pytorch I have losses that are 100x larger than in caffe. I think I use the same averaging over batches and pixels as in caffe but they are still different. The weird thing is that I in case of DispNetCorr1 disparities are not normalized, i.e they are in range from [0 ... 250] so I expect to get high losses in the begging of training (in order of 10-100), but they are still small in caffe log.

I use flownets_EPE1.951.pth.tar to test the examples below via run_inference.py

And get result like this

The image size of the result is 128 * 96 just 1/16 of the original image. And it does not look like the ground truth in Readme.md. Is there some thing wrong?

Hi, there. Thanks for the great code. I was trying to play with it on flying chair dataset. But the EPE in the end is around 5.0, which is far from 2.7 as stated in the original paper. So did you try to train the code from scratch and what performance do you have? Maybe I did something wrong. Thanks a lot.

Hey!

Ich checked your data augmentation. In random rotate there is

#flow vectors must be rotated too! careful about Y flow which is upside down
        target_=np.array(target, copy=True)
        target[:,:,0] = np.cos(angle1_rad)*target_[:,:,0] + np.sin(angle1_rad)*target_[:,:,1]
        target[:,:,1] = -np.sin(angle1_rad)*target_[:,:,0] + np.cos(angle1_rad)*target_[:,:,1]

but in RandomCropRotate it is

#flow vectors must be rotated too!
        target_=np.array(target, copy=True)
        target[:,:,0] = np.cos(angle1_rad)*target_[:,:,0] - np.sin(angle1_rad)*target_[:,:,1]
        target[:,:,1] = np.sin(angle1_rad)*target_[:,:,0] + np.cos(angle1_rad)*target_[:,:,1]

I guess the first one is correct? Furthermore, if positive y flow is pointing downwards wouldn't you have to change the translation as well? In RandomTranslate you do:

target[:,:,1]+= th

I guess if th is positive (you tranlsate upwards), your y flow would decrease?

What are your test scores on sintel (final, clean) or kitti? Many thanks

I downloaded the model in the pytorch path,but I can not open the tar file. #tar -xvf ***.tar tar: This does not look like a tar archive tar: Skipping to next header tar: Exiting with failure status due to previous errors

Sorry to bother you again. I just want to use the trained model to predict the optical flow. Could you share me the command only to predict. Thanks very much.

Hi, I try to use this implementation in my term project. However, I use the PIV image dataset instead of flying chairs. This dataset is a bit different from flying chairs. Image size is 256x256 and images are monochrome instead RGB. I took them from https://github.com/shengzesnail/PIV_dataset. I try to implement modifications mentioned in this article(https://link.springer.com/article/10.1007%2Fs00348-019-2717-2) on FlowNetS. However, I really have a problem with the data augmentation part. I got many errors especially the dimensionality part. I had already modified the architecture. I modified model input/output channels and filter sizes, but dataset loading parts unfortunately too complicate for me. I used the flyingchairs.py file with changing image format from .ppm to .tif but it doesn't work well so far. Do you have any advice? I really stuck in here. Thank you for your help. I attached an error example that I got.

Code training used in the process: loss = multiscaleEPE(output, target, weights=args.multiscale_weights, sparse=args.sparse) flow2_EPE = args.div_flow * realEPE(output[0], target, sparse=args.sparse) I can't understand the loss function you designed, could you please explain it?

At the bottom of README.md, there is a formula for ratate function which I think has a little typo. I suppose the correct formula is:

rotate: x,y,theta ->  (x*cos(theta) - x*sin(theta), y*cos(theta) + x*sin(theta))

, where the second element of output is y*cos(theta) + x*sin(theta) rather than y*cos(theta), x*sin(theta).

Finally, thank you so much for this nice implementation of flownet !

When I try to run inference on my images with run_inference.py and a pre-trained model load in the google drive mentionned for it : https://drive.google.com/drive/folders/0B5EC7HMbyk3CbjFPb0RuODI3NmM I have this error :

Traceback (most recent call last):
  File "run_inference.py", line 118, in <module>
    main()
  File "/home/abaldanza/anaconda3/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "run_inference.py", line 80, in main
    network_data = torch.load(args.pretrained)
  File "/home/abaldanza/anaconda3/lib/python3.8/site-packages/torch/serialization.py", line 593, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "/home/abaldanza/anaconda3/lib/python3.8/site-packages/torch/serialization.py", line 762, in _legacy_load
    magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, '\x04'. 

As if the pre-trained model can't be used i think ... If anyone knows why or how to fix it (if there is anything to do with de model) i should be in

In the main.py file,there is a Normalization operation which is " transforms.Normalize(mean=[0.45, 0.432, 0.411], std=[1, 1, 1])". I was confused of the meaning of the values [0.45, 0.432, 0.411]. I guess they are The mean of the dataset, but why they are fixed for different datasets? Moreover, the values are in the opposite order in the main.py file and run_inference.py file, However, I think they should keep consistent.