The forward warping functions in softsplat.py produce warped output only when the device id is 'cuda:0'. With other GPUs, the forward warped output is the same as the initialized zero tensor. Is there an approach to perform the forward warp in GPU devices other than the one with device id 'cuda:0'?

I tried to warp the second image back to the first one based on the ground truth flow, from the MPI-Sintel training split, to visualize the effect of the flow filed. From my understanding, the warped image should be similar to the first one. However, I can see that warped one is a kind of overlapped images of both frame1 and frame2. Top left: frame 1; Top right: frame 2; Bottom Left: warped frame

Indicated by the papers of yours and PWC-Net. It's easy to understand that the overlapped area is caused by those pixels appearing in the first frame but disappearing in the second frame.

Finally, here is my question: How could you use such an "inaccurate" flow from PWC-net to warp the frames "accurately"? Do those "inaccurate" flow really help you interpolate?? As you know, MPI-Sintel provides the occlusion mask to handle the problem above so that the final loss could be regularized by that mask. However, interpolation datasets don't have such a mask.

Thank you for your nice work! The bilinear kernel in your paper is descriped as followed: As far as I know, bilinear operation is used on four regular two-dimensional points, and how to used it on the an irregular and arbitrary number of warped points? Please explain this problem, thanks.

Hello, I am using the average forward warp on multiple GPUs, but I encountered one terrible error which is: File "xxx/softsplat.py", line 354, in FunctionSoftsplat tenNormalize[tenNormalize == 0.0] = 1.0 RuntimeError: CUDA error: an illegal memory access was encountered

I am quite confused that it should be one change value on one tensor but it caused illegal memory access error. Could you please help me with it? it happens after several epochs

Thank you for your insightful work!

When I train a network (different architecture to your paper) using forward warping, I found that the model crashed when fine-tuning together with PWC-Net (reflected by both the training loss and validation PSNR). The interpolation network is trained before fine-tuning with PWC-Net.

To check the problem, I tried a very simple model: Given two images x1, x2,

1. Compute the flow from x1 to x2, and multiply by 0.5.
2. Forward warp x1 using average splatting (to disregard the effect from the metric).

In this model, only the PWC-Net is trainable. However, the performance is strictly worse than using the pretrained PWC-Net. I have tried different learning rate and none of them works.

Do you have any ideas about the problem? Thanks again!

Hello, niklaus:

When I try to add the beta to the network, after training for a few epoches, it turn out to be None, the learning rate was initially set to 0.001, is it convenient for you to give me some advice of this part?

Looking forward for your reply~

Wen

Hi,

In the paper, it mentions that the model is ~31MB, but the official PWC-Net itself has ~41MB model. Is the model size measured in the paper does not include the PWC-Net?

I used the same method as #5 and got a similar result (PSNR:34.95). Therefore, according to your suggestion, I added different weights to each layer in the pyramid. Now the loss function I use is as follows:

class LaplacianPyramid(nn.Module):
    def __init__(self, max_level=5):
        super(LaplacianPyramid, self).__init__()
        self.gaussian_conv = GaussianConv()
        self.max_level = max_level

    def forward(self, X):
        t_pyr = []
        current = X
        for level in range(self.max_level):
            t_guass = self.gaussian_conv(current)
            t_diff = current - t_guass
            t_pyr.append(t_diff)
            current = F.avg_pool2d(t_guass, 2)
        t_pyr.append(current)

        return t_pyr

class LaplacianLoss(nn.Module):
    def __init__(self):
        super(LaplacianLoss, self).__init__()

        self.criterion = nn.L1Loss()
        self.lap = LaplacianPyramid()

    def forward(self, x, y):
        x_lap, y_lap = self.lap(x), self.lap(y)
        weights= [1, 2, 4, 8, 16, 32]
        return sum(weights[i] * self.criterion(a, b) for i, (a, b) in enumerate(zip(x_lap, y_lap)))

But I got a worse result. Could you please tell me how to change my loss function. 2. in Table 1 (Ablation experiments to quantitatively analyze the effect of the different components of our approach) of the paper, is the difference between Ours-CtxSyn-like and Ours-1 feature level only the difference between feature extractors(the feature extractor of Ours-CtxSyn-like is ResNet-18-conv1 and the one of Ours-1 feature level is cond2d->PReLU->cond2d->PReLU)?

Hi,

I face the following error when I run the code:

 input = input.contiguous(); assert(input.is_cuda == True)

TabError: inconsistent use of tabs and spaces in indentation

Thansk for the great work! However, it seems the gradient for |ux| < 1 in Equation (10) is incorrect?

We are taking gradient of flow_x_at_q w.r.t. 1 - |u_x|, say \partial{1 - |u_x|} / \partial{F_x}, and u_x = p_x - (q_x + F_x_at_q)

Then it should be \partial{1 - |u_x|} / \partial{u_x} * \partial{u_x} / \partial{F_x}.

\partial{1 - |u_x|} / \partial{u_x} = - sgn(u_x) while \partial{u_x} / \partial{F_x} = -1. Thus shouldn't the gradient be +sgn(u_x) rather than -sgn(u_x)? I'm wondering whether this is the factor leading to failure for finetuning.

Thanks!

In the neural scene flow fields paper you use 3d splatting to compute time interpolation views. I wonder how exactly it is done, do you splat the 3d points themselves, or do you project them onto the image plane first then splat the 2d pixels?

It seems that the current implementation only supports 4D tensors (i.e. 2d images splatting).

Hello, well done on your new paper, I found it really interesting! Do you plan on releasing the source code of the corrected softmax-splatting operator and the new pipeline?