Hi,

Thanks for your amazing work!! I am currently trying to reproduce train.py (mainly copy from RAFT project). Some training details are listed as following:

1. learning rate: 1e-4.
2. epochs: 40.
3. loss: sequence loss from RAFT (https://github.com/princeton-vl/RAFT/blob/master/train.py#L47) with gamma equals to 0.8.
4. random crop size [288, 384]

However, the performance is not satisfying even on training set (e.g. 1.207 epe on croped image/events and 2.913 epe on full image/events).

Did this happen to you while training? Looking forward to your reply.

The following are screenshots of my training status. Notice that 1px/3px/5px means the ratio of pixels whose epe is smaller than xx px compared with all valid pixels.

I'm trying to train the model using desc-flow dataset. While trying, I got some questions about it.

In this page (https://dsec.ifi.uzh.ch/dsec-datasets/download/), I could check that gt flow is consisted of png.

• png : ( 480, 640, 3 , uint8) _ like below
• Is it correct gt flow? Or Should i transform that png file for using in training?

And In 'dsec_loader.py', there is an function "load_flow" and I thought that function makes the flow & valid2d from png. But When i apply the training dataset(.png flow) to the function('load_flow'), I get the tensor filled with only 0. And also that function require uint16 data type, but the png file that i downloaded in above link is uint8.

So I want to ask : How can i generate the correct GT Flow & Mask(valid2d) from training dataset(.png flow)? And If i want to train the E-Raft model using dsec dataset, should i train the model by comparing the flow generated from png and predictions?

Also When i debugged the 'dsec_loader.py', flow_timestamp variable is entered the image timestamps in "image_timestamp.txt". Why should we put the image timestamp value in flow timestamp variable? Is it because the test dataset generate the gt flow as a image and measures the metric? If then, When i want to train the model using training dataset(dsec) in above link, should flow timestamp.txt value be entered as the flow timestamp variable?

Your reply will help me a lot. Thank you so much:)

Hi, thanks a lot for your work. I am confused about how to find the ground truth flow corresponding to events during training. The ground truth flows are give with timestamps, should I use the timestamps to relate them to events? And what's the usage of ' ms_to_idx' in the events.h5 file ? Can you give me some introductions and suggestions? Thanks a lot.

Hi，Mathias

Thanks for your cooperation!

The problem about the calibration between the event camera and the frame camera has solved with your help! Thanks again!

The test code for the E-RAFT has published in github, it’s a wonderful work!

In addition, I’d like to ask some questions about training the model for estimating the optical flow on DSEC dataset.

1. During supervision training on the model for E-RAFT, how to preprocess the ground truth of optical flow.

2. Currently, the network I proposed can converge on MVSEC dataset and get reasonable results. But it cannot converge on the DSEC dataset with the same network and the loss falls into local optimizaiton. I have tried many solutions (BN, Data Augmentation and so on), but still don’t work. What’s the reason do you think cause this problem.

3. How to prepare the trianing data of DSEC for optical flow estimation. My current approach is that choose the RGB frame and events according to the timestamps of optical flow maps. For example, the t1 and t2 is the timestamps of the optical flow map. Then the RGB_frame_1 and RGB_frame_2 which the t1 and t2 in theis exposure timestamps respectively. As for choose the events, just run the code that “event_data = eventslice.get_events(t1,t2)”.

I am not sure if the appoach mentioned above is correct.

Looking forward to receiving your reply!

Best regards,

Chi

Hi Mathias, Thank you for publishing this project. It is very helpful.

I am trying to do the data manipulation of the dataset but I found that you have removed some parts manually since they are static. I found that my results for training is not good at all, and I suspect doing something wrong in this truncation part.

My method works as follows:

1. I compare the elements one by one from both lists "file" and "self.timestamps_flow" from class "Sequence".
2. Since I compare after you created "self.timestamps_flow", When I find mismatch between the two lists, I remove the current element from "self.timestamps_flow" then continue deleting elements from it until I find the current element from both lists are equal, and so on.
3. For some sequences, I found that a part at the end of the file should be deleted as well, so I wrote an extra condition that in case that I reach the end of the "file" list, then I delete the rest of elements from "self.timestamps_flow" after that index.

After applying the above algorithm, I had more than 8000 samples, but you mentioned that they are 7800 samples only.

I would be grateful if you enlightened me with what might be the problem.

Thanks in advance

Hi，Mathias

There are 4 tables in the paper showing the experiment results. I assume they are all evaluation results.

The table 2 and table 3 are the MVSEC results. In my opinion, the EV-FLOW and E-RAFT are trained and evaluated with MVSEC dataset. Am I right?

Are results in table 4 achieved by training EV-FLOW and E-RAFT both on DSEC dataset ? Or are they just trained on MVSEC but evaluated with DSEC？If they are trained and evaluated both on DSEC, why are the results of E-RAFT not good as the results trained on MVSEC?

The ablation study are trained and evaluated separately on DSEC and MVSEC, right ?

I hope that I express my problem clearly.

Hi Mathias, I notice that you trained EV-FlowNet on DSEC for Comparison， but I am not sure the parameters setting. I want to know if the data setting is always consistent with the one used on training E-RAFT. (e.g. the voxel bins ==15 in loader-dsec.py)

By the way, since the GT flow in test dataset is unavailable, I randomly split the training set for training and testing, and found that the domain gap caused by day/night sequential difference is obvious. Does that influence the training result a lot ?

Hello, I'm implementing the training code of E-RAFT. But there's a sequence_length variable in the paper and code, so what exactly does that mean?

If i set a sequence_length = 2, it seemed like 2 sequential data were being loaded on the code. ( like below ) e.g) [ 0, 1 ]

If so, by any chance, Can i think of sequence_length as the number of applying warm_start at once?

For example, if i set the batch_size = 2 & sequence_length = 5, the sequence data will be loaded by 2 batches of 5 each like below. 1 batch : [0, 1, 2, 3, 4] 2 batch : [10, 11, 12, 13, 14] In each batch, the first "flow_init" variable will be initialized to 0, then warm start will be applied.

Is it right way that i understood it now?

Thank you for your great work!!

Hi，Mathias

I want to train the E-RAFT on the MVSEC dataset but I don't know how to pre-process the dataset .

I downloaded the MVSEC Outdoor Day 1 for 20 Hz evaluation that you released, but it is so different from the original data set. I don't know how to split the data into so many files and how to generate the 3 different timestamps_XXX.txt files.

Would it be possible for you to release the code for processing it ?

Thanks a lot.

Greetings, I am trying to load my checkpoint in the test code to get the results and submit on the benchmark but I encountered some problems, I would be grateful for your help.

1. I found that the test code saves in two folders called "submission" and "visualizations" and I followed the code and found that you don't save a flow file in the "submission" folder for each sample of the input sequences, does this mean that evaluation in the benchmark doesn't use all of the GT flow of the test sequences?

2. I was trying to run the test code on my GPU but I got an error in. I debugged the error and found that using the GPU will use the following line from test.py in the _test function batch = self.move_batch_to_cuda(batch) After this line, the variables in the batch changes from int to float, which resulted in having the following values for the first sample in the function "visualize_events" in visualization.py when using the cpu:

The t_start_us is: 49740000566 The t_end_us os: 49740100566

and in case I used the GPU, those values became:

The t_start_us is: 49740001280.0 The t_end_us os: 49740101280.0

which results in an error at the last sample of the first sequence in the function "visualize_flow_colours" in visualization.py.

I am using the latest version of the test code, but I added only the following line in the main.py due to having some error:

os.environ["GIT_PYTHON_REFRESH"] = "quiet"

Do you have an idea about what might be the problem? is it necessary to use the CPU?

3. Is it better any way in having the batch number greater than 1 while running the test code?

4. I am getting bad results when I load my checkpoint, the flow images in the submission folder are like the one below:

I am not sure about what the problem might be, given that the validation and training epe graphs are very good. I save the whole model in my checkpoint in the train code (with extension .pth) and then load it back in the test code. Do you have any advice regarding this?

Thanks in advance for your help

Hi，Mathias

I noticed that there are no ground truth flow in the evaluation or test sequences of DSEC dataset, and some training sequences neither. Did you just split the sequences with ground truth into training and test sequence to get the EPE score?

Is there any plan to release the source code for training? It will be greatly beneficial for the community and later works for a comparison in the future.