Hi, thank you so much for your great work and open source. I am very interested in your research, including the previous papers I have read.

When I tried to run the source code of PGP, I found that the working efficiency of the GPU was very low, basically not more than 10%, which caused the training to be extremely slow.

I'm using an NVIDIA RTX2080 GPU, and my computing power is much lower than the computing platform used in your paper. I'm not sure if the low usage is due to GPU memory and I took the liberty to ask this question, thank you very much!

Thanks for your work! I can't download the trained model weight via the link you offered because of the restrictions of network permission. Could you offer other download channels. I will be infinitely grateful.

Hi

First of all, thank you for sharing the code, which is very helpful to the community!

When I'm reading your code, I found a major difference between the code implementation and the paper: the encoding process of surrounding agents.

• the paper Both raw feature of vehicles and pedestrians are s_t^i which including an indicator I^i with value 1 for a pedestrian and 0 for a vehicle

And you use the same GRU to encode both vehicles and pedestrians So there are 3 GRUs in total: one for lane nodes, one for target agent, and one for surrounding agents

• Code There is no indicator in the raw feature of vehicles and pedestrians And you use 2 independent GRUs to encode vehicles and pedestrians:

https://github.com/nachiket92/PGP/blob/eb24e1904d4eae15200a02089992718aeb7a8149/models/encoders/pgp_encoder.py#L47-L51

then cat them together: https://github.com/nachiket92/PGP/blob/eb24e1904d4eae15200a02089992718aeb7a8149/models/encoders/pgp_encoder.py#L119

So there are 4 GRUs in total: one for lane nodes, one for target agent, one for surrounding vehicles and one for surrounding pedestrians

They are two different encoding methods, have you tried both? And what's the reason for choosing the latter?

Many greetings

Thanks for your excellent work!

Unfortunately, when I run the training code, I get some errors:

[libprotobuf FATAL google/protobuf/stubs/common.cc:83] This program was compiled against version 3.9.2 of the Protocol Buffer runtime library, which is not compatible with the installed version (3.20.1).  Contact the program author for an update.  If you compiled the program yourself, make sure that your headers are from the same version of Protocol Buffers as your link-time library.  (Version verification failed in "bazel-out/k8-opt/bin/tensorflow/core/framework/tensor_shape.pb.cc".)
terminate called after throwing an instance of 'google::protobuf::FatalException'
  what():  This program was compiled against version 3.9.2 of the Protocol Buffer runtime library, which is not compatible with the installed version (3.20.1).  Contact the program author for an update.  If you compiled the program yourself, make sure that your headers are from the same version of Protocol Buffers as your link-time library.  (Version verification failed in "bazel-out/k8-opt/bin/tensorflow/core/framework/tensor_shape.pb.cc".)
*** SIGABRT received at time=1652294159 on cpu 2 ***
PC: @     0x7f98057c6e87  (unknown)  raise
    @     0x7f9805b8b980  (unknown)  (unknown)
    @ 0x736177206d617267  (unknown)  (unknown)
[2022-05-11 20:35:59,564 E 25630 25630] logging.cc:325: *** SIGABRT received at time=1652294159 on cpu 2 ***
[2022-05-11 20:35:59,564 E 25630 25630] logging.cc:325: PC: @     0x7f98057c6e87  (unknown)  raise
[2022-05-11 20:35:59,566 E 25630 25630] logging.cc:325:     @     0x7f9805b8b980  (unknown)  (unknown)
[2022-05-11 20:35:59,569 E 25630 25630] logging.cc:325:     @ 0x736177206d617267  (unknown)  (unknown)
Fatal Python error: Aborted

Stack (most recent call first):
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1043 in create_module
  File "<frozen importlib._bootstrap>", line 583 in module_from_spec
  File "<frozen importlib._bootstrap>", line 670 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 967 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 983 in _find_and_load
  File "/media/work2/lishijie/miniconda/envs/pgp/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 64 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 728 in exec_module
  File "<frozen importlib._bootstrap>", line 677 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 967 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 983 in _find_and_load
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1035 in _handle_fromlist
  File "/media/work2/lishijie/miniconda/envs/pgp/lib/python3.7/site-packages/tensorflow/python/pywrap_tfe.py", line 28 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 728 in exec_module
  File "<frozen importlib._bootstrap>", line 677 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 967 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 983 in _find_and_load
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1035 in _handle_fromlist
  File "/media/work2/lishijie/miniconda/envs/pgp/lib/python3.7/site-packages/tensorflow/python/eager/context.py", line 35 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 728 in exec_module
  File "<frozen importlib._bootstrap>", line 677 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 967 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 983 in _find_and_load
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1035 in _handle_fromlist
  File "/media/work2/lishijie/miniconda/envs/pgp/lib/python3.7/site-packages/tensorflow/python/__init__.py", line 40 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 728 in exec_module
  File "<frozen importlib._bootstrap>", line 677 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 967 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 983 in _find_and_load
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 953 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 983 in _find_and_load
  File "/media/work2/lishijie/miniconda/envs/pgp/lib/python3.7/site-packages/tensorflow/__init__.py", line 41 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 728 in exec_module
  File "<frozen importlib._bootstrap>", line 677 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 967 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 983 in _find_and_load
  File "/media/work2/lishijie/miniconda/envs/pgp/lib/python3.7/site-packages/tensorboard/compat/__init__.py", line 45 in tf
  File "/media/work2/lishijie/miniconda/envs/pgp/lib/python3.7/site-packages/tensorboard/lazy.py", line 50 in load_once
  File "/media/work2/lishijie/miniconda/envs/pgp/lib/python3.7/site-packages/tensorboard/lazy.py", line 97 in wrapper
  File "/media/work2/lishijie/miniconda/envs/pgp/lib/python3.7/site-packages/tensorboard/lazy.py", line 65 in __getattr__
  File "/media/work2/lishijie/miniconda/envs/pgp/lib/python3.7/site-packages/tensorboard/summary/writer/event_file_writer.py", line 72 in __init__
  File "/media/work2/lishijie/miniconda/envs/pgp/lib/python3.7/site-packages/torch/utils/tensorboard/writer.py", line 62 in __init__
  File "/media/work2/lishijie/miniconda/envs/pgp/lib/python3.7/site-packages/torch/utils/tensorboard/writer.py", line 252 in _get_file_writer
  File "/media/work2/lishijie/miniconda/envs/pgp/lib/python3.7/site-packages/torch/utils/tensorboard/writer.py", line 221 in __init__
  File "train.py", line 33 in <module>
Aborted (core dumped)

However, I configure the environment according to the README, and if I directly install protobuf==3.9.2 there still exists other conflicts make the code cannot run.

My environment is:

Package                      Version
---------------------------- -------------------
absl-py                      1.0.0
aiosignal                    1.2.0
argon2-cffi                  21.3.0
argon2-cffi-bindings         21.2.0
astunparse                   1.6.3
attrs                        21.4.0
backcall                     0.2.0
beautifulsoup4               4.11.1
bleach                       5.0.0
cached-property              1.5.2
cachetools                   5.0.0
certifi                      2021.10.8
cffi                         1.15.0
charset-normalizer           2.0.12
click                        8.1.3
cycler                       0.11.0
debugpy                      1.6.0
decorator                    5.1.1
defusedxml                   0.7.1
descartes                    1.1.0
distlib                      0.3.4
entrypoints                  0.4
fastjsonschema               2.15.3
filelock                     3.6.0
fire                         0.4.0
flatbuffers                  2.0
fonttools                    4.33.3
frozenlist                   1.3.0
gast                         0.3.3
google-auth                  2.6.6
google-auth-oauthlib         0.4.6
google-pasta                 0.2.0
grpcio                       1.43.0
h5py                         2.10.0
idna                         3.3
importlib-metadata           4.11.3
importlib-resources          5.7.1
ipykernel                    6.13.0
ipython                      7.33.0
ipython-genutils             0.2.0
ipywidgets                   7.7.0
jedi                         0.18.1
Jinja2                       3.1.2
joblib                       1.1.0
jsonschema                   4.5.1
jupyter                      1.0.0
jupyter-client               7.3.1
jupyter-console              6.4.3
jupyter-core                 4.10.0
jupyterlab-pygments          0.2.2
jupyterlab-widgets           1.1.0
keras                        2.8.0
Keras-Preprocessing          1.1.2
kiwisolver                   1.4.2
libclang                     14.0.1
Markdown                     3.3.7
MarkupSafe                   2.1.1
matplotlib                   3.5.2
matplotlib-inline            0.1.3
mistune                      0.8.4
mkl-fft                      1.3.1
mkl-random                   1.2.2
mkl-service                  2.4.0
msgpack                      1.0.3
nbclient                     0.6.3
nbconvert                    6.5.0
nbformat                     5.4.0
nest-asyncio                 1.5.5
notebook                     6.4.11
numpy                        1.18.5
nuscenes-devkit              1.1.9
oauthlib                     3.2.0
opencv-python                4.5.5.64
opt-einsum                   3.3.0
packaging                    21.3
pandocfilters                1.5.0
parso                        0.8.3
pexpect                      4.8.0
pickleshare                  0.7.5
Pillow                       9.1.0
pip                          21.2.2
platformdirs                 2.5.2
positional-encodings         4.0.0
prometheus-client            0.14.1
prompt-toolkit               3.0.29
protobuf                     3.20.1
psutil                       5.9.0
ptyprocess                   0.7.0
pyasn1                       0.4.8
pyasn1-modules               0.2.8
pycocotools                  2.0.4
pycparser                    2.21
Pygments                     2.12.0
pyparsing                    3.0.9
pyquaternion                 0.9.9
pyrsistent                   0.18.1
python-dateutil              2.8.2
PyYAML                       6.0
pyzmq                        22.3.0
qtconsole                    5.3.0
QtPy                         2.1.0
ray                          1.12.0
requests                     2.27.1
requests-oauthlib            1.3.1
rsa                          4.8
scikit-learn                 1.0.2
scipy                        1.4.1
Send2Trash                   1.8.0
setuptools                   61.2.0
Shapely                      1.8.2
six                          1.16.0
soupsieve                    2.3.2.post1
tensorboard                  2.8.0
tensorboard-data-server      0.6.1
tensorboard-plugin-wit       1.8.1
tensorflow                   2.3.0
tensorflow-estimator         2.3.0
tensorflow-gpu               2.3.0
tensorflow-io-gcs-filesystem 0.25.0
termcolor                    1.1.0
terminado                    0.13.3
tf-estimator-nightly         2.8.0.dev2021122109
threadpoolctl                3.1.0
tinycss2                     1.1.1
torch                        1.11.0
torchaudio                   0.7.0a0+a853dff
torchvision                  0.8.0a0
tornado                      6.1
tqdm                         4.64.0
traitlets                    5.2.0
typing_extensions            4.2.0
urllib3                      1.26.9
virtualenv                   20.14.1
wcwidth                      0.2.5
webencodings                 0.5.1
Werkzeug                     2.1.2
wheel                        0.37.1
widgetsnbextension           3.6.0
wrapt                        1.14.1
zipp                         3.8.0

Do you know how to solve this issue? Thanks!

Hello these days I've tried this code many times But when I run the train.py it always crashed at the beginning I follow the all instructions in your README.md So I'm confused if there is anything I need to setting?

I run this command and use the original .yml file without modified Hope you can help me~ Many thanks python train.py -c configs/pgp_gatx2_lvm_traversal.yml -r /media/ee904/Data_stored/data/Nuscenes -d /media/ee904/Data_stored/data/Nuscenes/preprocess -o /media/ee904/Data_stored/PGP/results -n 100

Hello thank you for the great work I have some problems when I try to evaluate the metrics. I use the CUDA11.0 and the pytorch 1.7.1 conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 cudatoolkit=11.0 -c pytorch

And I download the trained weights

I load the weight as -w /media/ee904/Data_stored/PGP/PGP_lr-scheduler/archive/data.pkl then the error pop up I think it's about the weight format.. Can you help me? Or should I need to use the older version of CUDA?

Hi, thank you so much for sharing great work.

I have some questions.

1. Have you ever tested this model on a real vehicle? If a model trained using data sampled at 2 Hz is loaded on a real test vehicle, real-time performance cannot be guaranteed. Is there any recommended method to solve this problem?

2. How can I change from single agent prediction to multi agent prediction? I wonder if it can be easily changed or if another process is required. If not, do you have any plans to do this?

Thanks for your time.

when i want to run the third step 3. Run the following script to extract pre-processed data. This speeds up training significantly.

python preprocess.py -c configs/preprocess_nuscenes.yml -r path/to/nuScenes/root/directory -d path/to/directory/with/preprocessed/data

it turns out to be segmentation fault and i check that when from positional_encodings import PositionalEncodingPermute2D it come up with segmentation fault (core dumped)

but when i import positional_encodings in the command line it turns out to be right.

I want to train model with Latent var only mode.

from issue #13 I changed train config but i got this Exception

Do you know what the problem is ?

====== Loading NuScenes tables for version v1.0-trainval... 23 category, 8 attribute, 4 visibility, 64386 instance, 12 sensor, 10200 calibrated_sensor, 2631083 ego_pose, 68 log, 850 scene, 34149 sample, 2631083 sample_data, 1166187 sample_annotation, 4 map, Done loading in 20.600 seconds.

Reverse indexing ... Done reverse indexing in 5.2 seconds.

Epoch (1/100) Traceback (most recent call last): File "train.py", line 38, in trainer.train(num_epochs=int(args.num_epochs), output_dir=args.output_dir) File "/media/han/289e8c3e-12db-47da-8b89-15b58bef567d/home/han/prediction_256/PGP/train_eval/trainer.py", line 109, in train train_epoch_metrics = self.run_epoch('train', self.tr_dl) File "/media/han/289e8c3e-12db-47da-8b89-15b58bef567d/home/han/prediction_256/PGP/train_eval/trainer.py", line 153, in run_epoch predictions = self.model(data['inputs']) File "/home/han/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call result = self.forward(*input, **kwargs) File "/media/han/289e8c3e-12db-47da-8b89-15b58bef567d/home/han/prediction_256/PGP/models/model.py", line 35, in forward outputs = self.decoder(agg_encoding) File "/home/han/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call result = self.forward(*input, **kwargs) File "/media/han/289e8c3e-12db-47da-8b89-15b58bef567d/home/han/prediction_256/PGP/models/decoders/lvm.py", line 57, in forward raise Exception('Expected ' + str(self.num_samples) + 'encodings for each train/val data') Exception: Expected 1000encodings for each train/val data

Thank you in advance.

Thanks for sharing the great work!

I was wondering how to try the Latent var (LV) only in Table 3: Decoder ablations of PGP paper. Is there any configuration?

Thanks in advance, Zhen

Hi!

I found that you retrieved indefinite future trajectory while generating the ground truth node sequence and look-up table of visited edges for training behavior cloning: https://github.com/nachiket92/PGP/blob/1dadaf3109212e14f34eb3eb20cf745dec5d7602/datasets/nuScenes/nuScenes_graphs.py#L285

As an experiment, I modified the line above to: fut_xy = self.helper.get_future_for_agent(i_t, s_t, 6, True) Nothing has changed except the time range, I just changed the length of the future trajectory to 6 seconds which is equal to the prediction horizon(t_f in preprocess_nuscenes.yml).

After this modification, I re-preprocessed the dataset and re-trained the model. Here are the metrics I got on nuScenes validation set with 6s for ground truth traversal:

MinADE_5: 1.29 MinADE_10: 0.95 MinFDE_1: 7.26 MissRate_5: 0.52 MissRate_10: 0.34

Thanks for sharing the code. I was wondering if the numbers reported here in the repo and also in the paper were on the validation dataset or for the test set ?

Thanks for the great work and the repository! I have trained the model from scratch and it yields similar results ( a bit worse but almost the same), but shouldn't we pre-train for 100 epochs and then finetune for other 100 as stated in the paper? If so, I think it would be good to indicate it in the README. It would also be good to indicate how to combine the different enc/agg/dec to reproduce the jobs in the benchmark, or what configurations are possible at all, as some aggregator outputs would not match some decoders - maybe providing different .yml files? Thank you!

Hi, I was trying to retrain PGP but I run into an issue with scikit-learn's KMeans implementation. Sometimes when the model tries to compute the Ward distances, it throws a broadcast exception for dists = wts * centroid_dists + np.diag(np.inf * np.ones(len(cluster_counts))) because the shapes of wts and centroid_dists are different.

The root cause seems to be that cluster_lbls and cluster_ctrs are inconsistent, so performing np.unique() for the cluster labels returns the wrong cluster_cnts. In scikit-learn's documentation, I notice the following

cluster_centers_ndarray of shape (n_clusters, n_features) Coordinates of cluster centers. If the algorithm stops before fully converging (see tol and max_iter), these will not be consistent with labels_.

May I ask how should I handle this exception?