Hi,

Thanks for your wonderful work on this paper, you guys did a good job!

I have a question: did you use the training set obtained from all cameras on the Nuscenes dataset?

Thanks ahead for your help!

Hi, I have noticed that in the code (https://github.com/Khrylx/AgentFormer/blob/cf13e4033ceef7bdcfbf27183842415e7b841d34/model/agentformer.py#L535) the center of both the past trajectory and the future trajectory is used to normalize the input data. However, the future trajectory should not be available in the test. Is this data snooping? Please let me know if there is anything wrong with my understanding. Many thanks!

As you described in section 3.2 of the paper: I can understand the purpose of the MSE term ||Y-\hat Y||^2 is to push the real value Y and the mean of the Guassian \hat Y as close as possible,because the Gaussian distribution has the maximum probability value at the mean value. But where did the weighting factor 1/(2beta) come from? Why dividing the variance by beta leads to this weighting factor?

This PR supports loading from and training on the Tepper dataset. This can only be used for pedestrian forecasting in the same way as ETH/UCY; there are no pose annotations. In a future PR, we will clean up the dataloader infrastructure and support running experiments on other datasets in a cleaner fashion.

Hello @Khrylx , thank you for your great work.

I didn't understand what is pred_epoch in cfg files for each dataset. Why it is different from a dataset to another in cfg files?

Thanks

Hi @Khrylx ,

Thank you for your work, it is really impressive.

I am wondering if you have the visualisation script for these visualisation that you did.

Hello, this is an excellent job, but I don't understand one question, I hope you can answer them. The first is why the eth/ UCY data processing needs to divid the scale: like "found_data = past_data[past_data[:, 1] == identity].squeeze()[[self.xind, self.zind]] / self.past_traj_scale". The self.past_traj_scale=2, but in Trajectron++, the dataset is not divide the scale.

Looking forward to your reply.

Hi, congrats on the excellent work. I was going through your code and found that while converting the position of the agent in the image to pixel position you are multiplying it with a scale = 3. That should mean that the distance between adjacent pixels is 1/3 m as opposed to 3 m mentioned in the paper. Please let me know if I am interpreting anything wrong.

Hi

I'm working on the nuScenes 1 sample training. After finishing 100 epochs of training of CVAE，I continued to train the trajectory sampler. But unfortulately I got NAN value during training on the diversity loss term.

I guess this is because the diversity loss term is divided by 0 when K=1，and It is meaningless to calculate the diversity loss when K=1

Maybe we need to modify this line: https://github.com/Khrylx/AgentFormer/blob/195aae0e466327ffff4e698e9747f30de33683e4/model/dlow.py#L24

In case I missed something: Have you ever encountered a NAN value when training trajectory sampler with K=1?

Hello, I have seen that you are using a mask on the MSE loss to not take into consideration the padded agents, which is good. However, why aren't you applying the same on the ADE and FDE metrics?

There is no velocity and heading available in eth and ucy dataset .If it is present in nuscenes data than where i can find it ? Why we need columns with -1.0 ? Thanks for the response !

Thanks for your great work ! I am trying to apply your model to continue learning .Without modify your network and dataloader , i encounter following error .Since it raise this error inside of your lib function , and there are some forward pass already , i find it very hard to debug.... Could you please help me out?

epoch:0,loss:mse: 20.561 (18.647) kld: 2.017 (2.326) sample: 20.089 (10.814) total_loss: 42.667 (31.787)
epoch:0,loss:mse: 20.943 (37.872) kld: 2.045 (2.312) sample: 20.115 (24.269) total_loss: 43.103 (64.452)
epoch:0,loss:mse: 21.206 (0.451) kld: 2.075 (2.000) sample: 20.142 (0.421) total_loss: 43.423 (2.872)
epoch:0,loss:mse: 20.992 (4.669) kld: 2.075 (2.000) sample: 19.957 (4.490) total_loss: 43.024 (11.158)
epoch:0,loss:mse: 21.082 (12.970) kld: 2.136 (6.585) sample: 20.000 (12.838) total_loss: 43.218 (32.393)
epoch:0,loss:mse: 21.042 (16.777) kld: 2.331 (3.709) sample: 19.912 (16.551) total_loss: 43.286 (37.037)
Traceback (most recent call last):
  File "/mnt/petrelfs/tangxiaqiang/code/trajnet/dev/./main.py", line 189, in <module>
    main()
  File "/mnt/petrelfs/tangxiaqiang/code/trajnet/dev/./main.py", line 162, in main
    task_iter(task, num_devices, pop, generation_id, loop_id, exp_config)
  File "/mnt/petrelfs/tangxiaqiang/code/trajnet/dev/mu2net_traj/main.py", line 83, in task_iter
    train_loop(paths, ds_train, ds_validation,
  File "/mnt/petrelfs/tangxiaqiang/code/trajnet/dev/./mytrain.py", line 203, in train_loop
    model_data = path.model()
  File "/mnt/petrelfs/tangxiaqiang/miniconda3/./lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/mnt/petrelfs/tangxiaqiang/code/trajnet/dev/./AgentFormer/model/agentformer.py", line 596, in forward
    self.inference(sample_num=self.loss_cfg['sample']['k'])
  File "/mnt/petrelfs/tangxiaqiang/code/trajnet/dev/./AgentFormer/model/agentformer.py", line 607, in inference
    self.future_decoder(self.data, mode=mode, sample_num=sample_num, autoregress=True, need_weights=need_weights)
  File "/mnt/petrelfs/tangxiaqiang/miniconda3/./lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/mnt/petrelfs/tangxiaqiang/code/trajnet/dev/./AgentFormer/model/agentformer.py", line 425, in forward
    self.decode_traj_ar(data, mode, context, pre_motion, pre_vel, pre_motion_scene_norm, z, sample_num, need_weights=need_weights)
  File "/mnt/petrelfs/tangxiaqiang/code/trajnet/dev/./AgentFormer/model/agentformer.py", line 341, in decode_traj_ar
    tf_out, attn_weights = self.tf_decoder(tf_in_pos, context, memory_mask=mem_mask, tgt_mask=tgt_mask, num_agent=data['agent_num'], need_weights=need_weights)
  File "/mnt/petrelfs/tangxiaqiang/miniconda3/./lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/mnt/petrelfs/tangxiaqiang/code/trajnet/dev/./AgentFormer/model/agentformer_lib.py", line 746, in forward
    output, self_attn_weights[i], cross_attn_weights[i] = mod(output, memory, tgt_mask=tgt_mask,
  File "/mnt/petrelfs/tangxiaqiang/miniconda3/./lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/mnt/petrelfs/tangxiaqiang/code/trajnet/dev/./AgentFormer/model/agentformer_lib.py", line 644, in forward
    tgt2, self_attn_weights = self.self_attn(tgt, tgt, tgt, attn_mask=tgt_mask,
  File "/mnt/petrelfs/tangxiaqiang/miniconda3/./lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/mnt/petrelfs/tangxiaqiang/code/trajnet/dev/./AgentFormer/model/agentformer_lib.py", line 506, in forward
    return agent_aware_attention(
  File "/mnt/petrelfs/tangxiaqiang/code/trajnet/dev/./AgentFormer/model/agentformer_lib.py", line 177, in agent_aware_attention
    raise NotImplementedError
NotImplementedError

Hi, I really like your work in dealing with multi-agent trajectories prediction. I went through the paper and codes and popped up a quick question about the time encoder. As you mentioned in the paper, the time encoder that integrated the timestamp features differs from the original positional encoder. But I cannot find the time encoder codes in this repo. Please let me know if I missed anything. Much appreciated!

Hi brother. I have installed your package, and all the environments have been configured, but I still can't run it, and such an error is reported. This is the file format and wrong statement.

Dear reader, thank you for the great work on this topic and for realising the code for the community to improve. I am currently trying to understand the data format however am very unclear about how the eth_ucy dataset is actually preprocessed. Do I understand it correctly that only x and y coordinates are used and no velocity and heading inofmration is extracted. As looking through the data most columns only contain -1 values. Could you provide a column name list for the inputs found in the datasets/eth_ucy files?

Furthermore, is my understadning correct that only the ego agent informatin is stored in a given row? As from reading the paper my understanding was that all agent states would be stored in a single entry for each timestep. Could you maybe elaborate on how you create the representation of the data that is described in the paper? Thank you!

Hi,

Thanks for sharing this great work!

I have a question about this paper, have you tried to get the most likely prediction on Nuscenes official prediction challenge and calculate the ADE?

Bo