Hi @AdrianZw

Are there any scripts or resources to convert the mobilestereonet to torchscript or ONNX format? I want to use those formats for inference in C++. Any suggestions would be greatly appreciated.

Thanks.

Hi there,

I just wanted to reach out to say that this looks awesome! We make a AI-capable stereo camera ecosystem called DepthAI (Github: https://github.com/luxonis/depthai-hardware) which actually is pretty well tuned to run MobileNetV1- and MobileNetV2- based networks.

So we are SUPER excited to try to get this running in our ecosystem!

No real issue here - just wanted to say thanks - and that we'll be working to get this running in DepthAI, issue https://github.com/luxonis/depthai/issues/476.

-Brandon \ OpenCV / Luxonis

I am trying to train MSNet3D on stereodriving and KITTI2015 but faced this error:

File "train.py", line 205, in train() File "train.py", line 98, in train loss, scalar_outputs, image_outputs = train_sample(sample, compute_metrics=do_summary) File "train.py", line 155, in train_sample disp_ests = model(imgL, imgR) File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "/opt/conda/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 161, in forward outputs = self.parallel_apply(replicas, inputs, kwargs) File "/opt/conda/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 171, in parallel_apply return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)]) File "/opt/conda/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 86, in parallel_apply output.reraise() File "/opt/conda/lib/python3.8/site-packages/torch/_utils.py", line 428, in reraise raise self.exc_type(msg) RuntimeError: Caught RuntimeError in replica 0 on device 0. Original Traceback (most recent call last): File "/opt/conda/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker output = module(*input, **kwargs) File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "/home/trainer/mobile_stereo/models/MSNet3D.py", line 124, in forward out1 = self.encoder_decoder1(cost0) File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "/home/trainer/mobile_stereo/models/MSNet3D.py", line 46, in forward conv5 = F.relu(self.conv5(conv4) + self.redir2(conv2), inplace=True) RuntimeError: The size of tensor a (4) must match the size of tensor b (3) at non-singleton dimension 2

Does anybody know how to solve this problem? Looking forward to your reply！

conda env create --file mobilestereonet.yml

... env created succesfully ...

(base) tornado@tornado:~/mobilestereonet$ conda activate mobilestereonet
(mobilestereonet) tornado@tornado:~/mobilestereonet$ ls
cost.py  datasets  filenames  images  LICENSE  mobilestereonet.yml  models  prediction.py  README.md  train.py  utils
(mobilestereonet) tornado@tornado:~/mobilestereonet$ python cost.py
Traceback (most recent call last):
  File "cost.py", line 28, in <module>
    from thop import profile
ModuleNotFoundError: No module named 'thop'

fixed by

pip install thop

HI. I get out of memory error when executing train.py code. I reduced batch size but get same error. my GPU is Nvidia 1060 6GB. the error is: RuntimeError: CUDA out of memory. Tried to allocate 32.00 MiB (GPU 0; 5.94 GiB total capacity; 4.78 GiB already allocated; 27.19 MiB free; 4.80 GiB reserved in total by PyTorch)

I appreciate any help to fix this error. thanks in advance.

update: I could successfully train a model with batch size 1 but with batch size greater than 1 I get "out of memory error" which is discussed above.

could you point me to where to download the right dataset for SF pretraining?

After follow Sceneflow hyperlink, I downloaded Sceneflow disparity and frames_finalpass dataset.

But when I checked ./filenames/sceneflow_train.txt, the file path are different from dataset:

./filenames/sceneflow_train.txt: for example: frames_finalpass/TRAIN/15mm_focallength/scene_backwards/slow/left/0224.png frames_finalpass/TRAIN/A/0605/left/0015.png frames_finalpass/TRAIN/B/0665/left/0006.png

in the downloaded dataset: for example: frames_finalpass/15mm_focallength/scene_backwards/slow/left/0224.png there is no TRAIN/, TRAIN/A/ or TRAIN/B directories.

Where is TRAIN/, TRAIN/A/, or TRAIN/B from? Is there a script that I can run to automatically download all the dataset needed and prepare them in the right directory hierarchy as specified in ./filenames/*.txt? Please advise.

Thank you very much for your help in advance.

e.g. https://github.com/cogsys-tuebingen/mobilestereonet/blob/main/filenames/ds_test.txt#L1 doesn't seem to match the filenames in DrivingStereo as in the dataset provided on the linked website: https://drivingstereo-dataset.github.io/, which, e.g., are in the form 2018-07-11-14-48-52_2018-07-11-15-45-53-241.jpg. also the folder structure seems to be different. it seems to be the same issue for SceneFlow. is there any mismatch in the version of the dataset(s) or preprocessing required? in case, is the preprocessing described somewhere (maybe i missed it?).

my device is 3080 16GB, but with the original code, I can only train with batch_size=1, otherwise, cuda out of memory.

I tried with Mixed Precision, and I can increase batch_size to 2, how can I train with large batch size without dropping accuracy.

Hi, i am using Cuda11.1 and encountered following error. RuntimeError: Legacy autograd function with non-static forward method is deprecated. Please use new-style autograd function with static forward method. (Example: https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function)

def make_nograd_func(func): def wrapper(*f_args, **f_kwargs): with torch.no_grad(): ret = func(*f_args, **f_kwargs) return ret return wrapper

This is the function that throws the error while calling func()

Thank you for your great work. I want to see the details of the the extension from 2D to 3D convolution and find there is no model ! I noticed a record of detete, is there something wrong? waiting for the rest, thank you again.

i want to verify other methods with yours，could you please share your model files？My mail address is [email protected]. Thanks a lot. @draeger

Hi,

I am impressed by your work and tried to run prediction on kitti dataset. It looks good.

But when I tried to run prediction on a monochrome dataset captured by an oak-d stereo camera, the results looks like random noise. Just wondering if you ever tried to run your model using monochrome input or not and how was the performance?

If you like I attached a pair of monochrome stereo images below for your trial. Please let me know how's your test result looks like. Thanks a lot

.

Does the maxdisp value has a influence on the result of inference time or the performance? Besides, the result does not work well on my own dataset, I guess it is the problem of maxdisp value.