Hello, thank you for introducing nice work and code!

I have a question about separable 4d convolution.

Is the WTA operation explicitly applied to the 4d separable convolution? In the code, I can't see the part of WTA in the conv4d.py. or is the WTA kernel in sec3.2 the same as the other kernel(e.g., spatial kernel)?

Plus, can you explain (or derive) more in details about the way to decompose 4d convolution into the wta and spatial convolution (sec3.2, [4d-convolution -> factorization]? Actually, it was hard to understand how the original 4d convolution be equal to the factorized one by equation.

It would be very appreciated if these questions are answered. Thank you!

Hi gengshan~~ Thanks for your wonderful work and code!

I have done the three training stage: FlyingChairs->FlyingThings->KITTI. I tried the following code to evaluate my model:

modelname=kitti-ft-trainval
i=149999
CUDA_VISIBLE_DEVICES=0 python submission.py --dataset 2015 --datapath /ssd/kitti_scene/training/   --outdir ./weights/$modelname/ --loadmodel ./weights/$modelname/finetune_$i.tar  --maxdisp 512 --fac 2
python eval_tmp.py --path ./weights/$modelname/ --vis no --dataset 2015

But I get the following error:

epe = np.sqrt(np.power(gtflow - flow,2).sum(-1))[mask]
ValueError: operands could not be broadcast together with shapes (370,1224,2) (375,1242,2)

Another problem is how to finetune my model on Sintel, I am confused with the "rob_devkit" tool. When I run python flow_devkit.py, it asks me which format to use. There are two choices: middlebury and kitti. Which format should I use? I tried to use kitti format, and get the following folder structure:

rob_devkit/
    flow/
        datasets_kitti2015/
            metadata/
            test/
            training/
            bundler/

But in your main.py, the sintel dataset is loaded by iml0, iml1, flowl0 = ls.dataloader('%s/rob_flow/training/'%args.database). I don't know where to find the rob_flow.

I am a newcomer to this field and puzzled by these problems for several days. Could you please help me?

Hi, thank you for your great work.

I have used the VCN module and it works well. But when I wanted to train the PWC-Net module to make a comparison, I found that just changing this(line 201, in main.py)

model = VCN([batch_size//ngpus]+data_inuse.datasets[0].shape[::-1], md=[int(4*(args.maxdisp/256)), 4,4,4,4], fac=args.fac)

to

model = PWCDCNet([batch_size//ngpus]+data_inuse.datasets[0].shape[::-1])

cannot work, it shows:

Iter 2 training loss = -7931635.000 , AEPE = 234449312.000 , time = 0.16
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1579027003190/work/aten/src/THC/THCCachingHostAllocator.cpp line=278 error=77 : an illegal memory access was encountered

I think it is because in models/PWCNet.py, line 300, the output is

return flow2*20,flow3*20,flow4*20,flow5*20,flow6*20,flow2, flow2[:,0]

It seems that he output[-2] and output[-1] are not the required loss and oor, so should I rewrite these part as that in VCN in order to train the PWCNet? namely, adding

            oor2 = F.upsample(oor2[:,np.newaxis], [im.size()[2],im.size()[3]], mode='bilinear')[:,0]
            oor3 = F.upsample(oor3[:,np.newaxis], [im.size()[2],im.size()[3]], mode='bilinear')[:,0]
            oor4 = F.upsample(oor4[:,np.newaxis], [im.size()[2],im.size()[3]], mode='bilinear')[:,0]
            oor5 = F.upsample(oor5[:,np.newaxis], [im.size()[2],im.size()[3]], mode='bilinear')[:,0]
            oor6 = F.upsample(oor6[:,np.newaxis], [im.size()[2],im.size()[3]], mode='bilinear')[:,0]
            loss += self.get_oor_loss(flowl0[:,:2]-0,        oor6, (64* self.flow_reg64.flowx.max()),occ_mask)
            loss += self.get_oor_loss(flowl0[:,:2]-up_flow6, oor5, (32* self.flow_reg32.flowx.max()),occ_mask)
            loss += self.get_oor_loss(flowl0[:,:2]-up_flow5, oor4, (16* self.flow_reg16.flowx.max()),occ_mask)
            loss += self.get_oor_loss(flowl0[:,:2]-up_flow4, oor3, (8* self.flow_reg8.flowx.max())  ,occ_mask)
            loss += self.get_oor_loss(flowl0[:,:2]-up_flow3, oor2, (4* self.flow_reg4.flowx.max())  ,occ_mask)

            return flow2*20, flow3*20,flow4*20,flow5*20,flow6*20,loss, oor2

Thank you very much for any help!

Hi gengshan~

Thanks for your wonderful work and code!

When I tried to train the VCNet according to the "Pretrain on flying chairs and flying things" part of readme, I found an error which is "one of the variables needed for gradient computation has been modified by an inplace operation". But I just modified some paths for my own environment, and didn't change other parts.

I am so confused about this error, so I wonder that: (1) have you met this error before? (2) can you give me some suggestions about solving this problem?

Thanks for your help! And I wish to hear from you soon!

Best wishes.

Blcony

Hi, to my understanding, flowl0 if of shape [H, W, 3], where the third channel indicates whether the flow at that position is valid. Specifically, mask_0 is used to track valid pixels after spatial transformation augmentation. However, I find that the target variable is always of two channels. So every time you use target[..., 2:3], you only get a numpy array of shape [H,W,0], which seems to be erroneous? https://github.com/gengshan-y/VCN/blob/00c4befdbdf4e42050867996a6f686f52086e01a/dataloader/flow_transforms.py#L206

Hi,

I'm trying to reproduce your results. In the README.md, finetuning on KITTI, sintel needs 1000 epoch.

Is a model trained for 1000epochs to reproduce the results in the paper?

thx.

Hi, I congrats for your work. I would like to know whether at inference time is it possible to use this model as part of a framework that performs the online action detection task? (i.e. let's suppose I have an input stream video, is the model able to compute optical-flow features as the frames arrive, with real-time speed ?)

Thank you!

Hi,

Thanks so much for open-sourcing your work!

I am trying to load your pre-trained model weights from (https://drive.google.com/drive/folders/1mgadg50ti1QdwfAf6aR2v1pCx-ITsYfE?usp=sharing) but I am not able to untar the finetune_149999.tar from neither "chairs" or "kitti-ft-trainval". Below is the error message:

tar -xvf finetune_149999.tar
tar: Error opening archive: Unrecognized archive format

Hi Gengshan,

You added the Chair, Things, Hd1k, and Kitti datasets to the Sintel dataset when fine-tuning. I guess this operation aims to improve generalization ability and avoid overfitting. I was wondering is this important for the Sintel benchmark result and training set performance?

Best Regards

Hi everyone! I am a beginner in Pytorch and Computer vision. I just cloned the source code(without installing the correlation module, I thought it is not necessary for training?) and began training using the command:

CUDA_VISIBLE_DEVICES=1,2 python main.py \
--maxdisp 256 --fac 1 \
--database /ssd2/ \
--logname chairs-0 \
--savemodel /ssd1/models/vcn/ \
--epochs 1000 --stage chairs --ngpus 2

and I got the following traceback:

File "main.py", line 258, in train
      vis['AEPE'] = realEPE(output[0].detach(), flowl0.permute(0,3,1,2).detach(),mask,sparse=False)
File "/home/VCN/utils/multiscaleloss.py", line 86, in realEPE
      return EPE(upsampled_output, target,mask, sparse, mean=True)
File "/home/VCN/utils/multiscaleloss.py", line 12, in EPE
      EPE_map = torch.norm(target_flow-input_flow,2,1)

Thanks very much for any help.

Hi Gengshan,

In the supplementary material of VCN, you highlight that the weight decay term in PWCNet is removed. May I ask what's the effect of removing it? Is it for faster training or performance boost?

Best Regards, Jianyuan

Hi,

Thank you for sharing your codes.

I am little confused that in your supplemental material for paper you said that you used L1 loss + OOR loss when finetuned Sintel and KITTI dataset.

However, in you code, it shows that you use L2 loss and OOR loss.

I would like to know which loss I should use.

https://github.com/gengshan-y/VCN/blob/00c4befdbdf4e42050867996a6f686f52086e01a/dataloader/flow_transforms.py#L186-L189 Hi, could you please explain what this exit condition means? thx!

Hi,

I am trying to reproduce your KITTI-15 validation (40 images) Fl-err and EPE result. I am using the finetune_149999.tar under kitti-ft-trainval. However I am unable to reporduce neither of your Fl-err (3.9) and EPE (1.144).

For EPE on 40 images, the average EPE I got is 2.527. As a reference, the average EPE is 2.5 for the kitti-train.

To calculate the EPE, I am using the following approach:

for index in range(len(test_left_img)):
    1. flow, _ = VCN(input)
    2. Resize flow to kitti Height and Width
    mask = gtflow[:,:,2]==1
    gtflow = gtflow[:,:,:2]
    flow = flow[:,:,:2]
    epe = np.sqrt(np.power(gtflow - flow,2).sum(-1))[mask]
     aepe_s.append( epe.mean() )

result = sum(aepe_s)/len(aepe_s)

Is this the right approach? Also, I don't seen to be able to find the code to compute Fl-err. Can you share this as well?

Thanks,

Hello, I am trying to run the example notebook. However, when I run this line flow,entropy = inference(imgL_o, imgR_o, maxdisp, fac, modelpath) I get this error:

~/Documents/GitHub/VCN/models/conv4d.py in forward(self, x)
    163 
    164     def forward(self,x):
--> 165         b,c,u,v,h,w = x.size()
    166         x = self.conv1(x.view(b,c,u,v,h*w))
    167         if self.with_bn:

ValueError: not enough values to unpack (expected 6, got 5)

This happens with both the sintel and KITTI models.