We prepare to cite your paper in our new work. When I reproduce your work, I noticed that the maximum learning rate of training Sceneflow is 0.0002 in your provided code. In your paper, It is said that Sceneflow is trained with a minimum learning rate of 1e-4. Should I just keep the parameter 0.0002 in the code to reproduce your work?

Thanks!

Thanks for sharing such an excellent work!
Mentioned in the paper:
Final models are trained on synthetic data for 200k steps with a batch size of 8
SceneFlow consists of about 35k training pictures.
Can we think that you have trained a total of 200k*8/35k ~= 45 epochs?
GANet, DSMNet, etc. only train 10-20 epochs. Have you compared the performance when the training epoch is less than 20?
How long does it take to train 200k Steps under your GPU configuration?

thank you for your reply!

Thanks for the great work Lahav and the team, iteratively refinement has been missing in stereo matching, and great work on the multi-level correlation lookup volume.

I have been using this code repo on my application with great success. However training is not very stable, the model seems to suffer some kind of mode collapse and predict the same for all inputs.

I'm just checking all the loose ends and came across the normalisation part. https://github.com/princeton-vl/RAFT-Stereo/blob/5c13878b617177da139cfeba79ac15b39b351963/train_stereo.py#L151

1. Why is batch norm frozen during training? Doesn't this defeat the purpose of adding a batch norm in the first place?
2. In the paper, instance norm is used instead of the batch norm for the context encoder, can you expand on this implementation detail? How will this impact the model when we use a shared encoder for speed up?

Thanks for sharing your excellent work! Why do you provide two different ways for augmenting data? The sparse one does not have yjitter，asymmetric color augmentation. Besides, why do you provide a new function resize_sparse_flow_map, instead of using cv2.resize(flow, interpolation=cv2.INTER_NEAREST)for resizing sparse gt?

When I converted the ONNX model, I encountered that the CorrSampler could not be converted. Can you provide some suggestions?

Looking forward to your reply！

Hi, thank you for sharing the awesome work. I have run the basemodel raftstereo-middlebury.pth without any refinement on my costume dataset. The precision of results are pretty good, But the runtime of model prediction seems does not match that described in the article. I add some time-analysis snippet in demo.py:

The command predicts stereo on custome data is : python demo.py --restore_ckpt models/raftstereo-middlebury.pth -l=output_xvisio/rect_cam0/*.jpg -r=output_xvisio/rect_cam1/*.jpg --corr_implementation alt --mixed_precision

The configuration of my local machine:

• image resolution: 640 x 400;
• CPU: Intel® Core™ i7-8700K CPU @ 3.70GHz × 12
• GPU: NVIDIA GeForce GTX 1080 Runtime of base model:

The configuration of my server machine:

• image resolution: 640 x 400;
• CPU:Intel(R) Xeon(R) CPU E5-2682 v4 @ 2.50GHz
• GPU: NVIDIA Tesla P100 Runtime of base model:

Could you help me to figure out what mistake I made caused this problem? I really appreciate your help!

Hi @lahavlipson, Thank you for your great work!

I'm wondering what datasets you used to train raftstereo-middlebury.pth? Why do you recommend it for in-the-wild images?

Hi, thank you for sharing the amazing work. It shows better performance on leaderboard with a large margin. I'm testing R-Stereo. I found that I can't make the speed of fastest model to reach ~26fps on kitti 2015 test data. So, I want to ask for the configuration to achieve this.

I use the time calculate method like this:

            padder = InputPadder(image1.shape, divis_by=32)
            image1, image2 = padder.pad(image1, image2)
            start_time = time.time()
            _, flow_up = model(image1, image2, iters=args.valid_iters, test_mode=True)
            print("forward time: ", time.time()-start_time)
            file_stem = imfile1.split('\\')[-1]

the configure is RTX3070 CUDA11 Cudnn 8 Win10 Kitti (1248x384) with following scripts

python.exe demo.py 
--restore_ckpt   models/raftstereo-realtime.pth
--shared_backbone
--n_downsample  3
--n_gru_layers  2
--slow_fast_gru
--mixed_precision
--corr_implementation  reg_cuda
-l= ${kitti_test}
-r= ${kitti_test}

I got about ~170ms per stereo image. I want to ask which GPU are you use and if there anything wrong with the testing?

Hi
I want to compute the up-down disparity , and then fusion with left-right disaprity, my solution is

disparity_x = RAFT_x(Img_Left, Img_Right) disparity_y = RAFT_y(Img_Up Img_Down) disparity_out = disparity_x * ratio_x + disparity_y * ratio_y

In RAFT_y I modify correction function to y-direction correction but result is not good,could you give me some advice?

Thanks

Hello, Is it possible to extract a confidence map for all disparity values? During a 3D stiching and SLAM such confidence map would greatly improve the over all result. Do you have any suggestions on how to extract such confidence map from the RAFT pipeline itself or would you recommend a different approach? Thank you for making your work available to all!

Hi
In paper section 4.4. Middlebury,After pre-training on Sceneflow [23], we fine-tune on 384x1000 random crops of the 23 Middlebury traning images for 4000 steps with a batch size of 2, using 22 update iterations during training but in official_train.txt, only containt 10 Middlebury traning images , could you help me point out which is right?

Thanks

https://github.com/princeton-vl/RAFT-Stereo/blob/0e2a12746143a7552e30ef2f4b1d4c3214388a1a/train_stereo.py#L222 Hi there, thank you for supplying such a clear code! I have a question regarding the training procedure; as I understand, the training you suggest on the git page includes only sceneflow (and refine on Middlebury) with no reference to other datasets - FallingThings and Tartanair, which you reference in the paper. Do you use them in any additional training? Can you clarify? Thank you!

/home/user/anaconda3/envs/raftstereo/lib/python3.7/site-packages/torch/optim/lr_scheduler.py:136: UserWarning: Detected call of lr_scheduler.step() before optimizer.step(). In PyTorch 1.1.0 and later, you should call them in the opposite order: optimizer.step() before lr_scheduler.step(). Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate

Does it have an impact on the results? If so, how to solve it？ Thanks！

https://github.com/princeton-vl/RAFT-Stereo/blob/0e2a12746143a7552e30ef2f4b1d4c3214388a1a/core/corr.py#L156 Hi, what does this line mean? In the paper it says that the correlation is dot product between feature vectors. But it is divided by this sqrt. Any meaning? Can replace this sqrt by something else?

Hi, RAFT-Stereo is producing disparities also in regions that are occluded to one of the cameras of the stereo pair. As a result the disparites are good when scene is similar to the training set. In terms of generalization it then produces more errors in those partially occluded regions that are less similar to data in the training set.

In SGM implementations the left-rigth and right-left consitency check is used to implicitly find occluded regions. This check is done in SGM within the cost cube computed within one run (only left-right or only right-left) by swithching the Cost look up direction.

Where in the RAFT-Stereo implementation would you suggest implementing an equivalent for occlusion check without having to implement the full disparity computation twice? Thanks.

Hi. I tried RAFT for my data. It's working fine. But objects in my case are thin and I'm facing depth issue (zig-zag kind structures). I saw in one issue you suggested to use High Resolution images or Retrain with thin dataset. As I don't have GT for my dataset. Can you suggest any opensource dataset that have more thin objects so I can ReTrain RAFT... Thanks!!!

Hi,

I realized that you excluded this subset from the tartan_air, could you share the reason why you did so? and why only "easy" not "hard"?

Thanks!