Thanks for providing the codes, it's amazing work. 🤩

Here are some questions after I read the paper and code readme:

1. Is the dataset provided in the repo the whole dataset as in the paper Table 7 which has 399776 frames?
2. Is the expert agent from lbc repo as the link is shown here? or as the paper said the CARLA behavior agent as link shown here? Since I didn't see the code for collecting the dataset.
3. As the dataset said in the paper is collected in all towns, as I knew leaderboard official public routes just have Town01-06 routes file? and I didn't find any additional routes files in this repo? would you mind to release that the route file you collected for data? since if we want to compare with the method it should have the same training routes for fair. If not, the all towns in the paper said, is that includes other towns like Town07 and Town10HD as said in paper table 7? or you also build another map for training?

Looking forward to your reply, and thanks again for this paper and codes.

Thanks for your fantastic work! I have read the paper and noticed that the mapping prediction and detection in perception training are both supervised. Since I didn't find details about data generation in the paper, I am really curious about how do you generate the labelled data for such a large scale dataset, or how do you get the labels for the collected data?

Looking forward for your reply. Thanks very much!

Question about the training phase:

1. The whole dataset has some data.mbd loss in folders, and the provided dataset frame is shown here:

   

2. At training phase: End-to-end Training, I downloaded the whole dataset and it's wried at some of the loss, I'd like to ask that is it correct or just normal for these kinds of loss?

   

   The training detail is same as the default config.yml with all phases are trained in 100 epochs and bev with 160 epochs with suitable batch size, the evaluation is really terrible at the online leaderboard shown I just finished the first five route and see the result file notice the collision is serious, I'm wondering is there any process I miss that cannot reproduce the effect to evaluate? since with only 1% data loss will not effect so much on the training model.

Thanks again for your work!

Thank you for your amazing work!

I have followed INSTALL.md to setup the environment . When I runROUTES=assets/routes_lav_valid.xml ./leaderboard/scripts/run_evaluation.sh , the following error occurs.

========= Preparing RouteScenario_0 (repetition 0) =========

Setting up the agent

Could not set up the required agent:

invalid load key, 'v'.

Traceback (most recent call last): File "/home/abc/carla1/LAV-main/leaderboard/leaderboard/leaderboard_evaluator.py", line 262, in _load_and_run_scenario self.agent_instance = getattr(self.module_agent, agent_class_name)(args.agent_config) File "/home/abc/carla1/LAV-main/leaderboard/leaderboard/autoagents/autonomous_agent.py", line 45, in init self.setup(path_to_conf_file) File "/home/abc/carla1/LAV-main/team_code/lav_agent.py", line 89, in setup bev_planner.load_state_dict(torch.load(self.bev_model_dir)) File "/home/abc/anaconda3/envs/LAV-env/lib/python3.7/site-packages/torch/serialization.py", line 713, in load return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args) File "/home/abc/anaconda3/envs/LAV-env/lib/python3.7/site-packages/torch/serialization.py", line 920, in _legacy_load magic_number = pickle_module.load(f, **pickle_load_args) _pickle.UnpicklingError: invalid load key, 'v'.

Registering the route statistics

ps: import torch torch.version '1.11.0'

any help will be appreciated!

Thanks to the author for the wonderful code! I would like to ask how to visualize the results of the network output, for example: RGB camera inputs, predicted road geometries, and detection and motion predictions respectively.

Looking forward to your reply, thank you very much!

Hi,

I got error when I ran ROUTES= ./leaderboard/scripts/run_evaluation.sh in LAV-env conda environment. Here is the error message. I am running Ubuntu 18.04 with the installation steps followed. Traceback (most recent call last): File "/home/demo/Public/LAV/leaderboard/leaderboard/leaderboard_evaluator.py", line 457, in main leaderboard_evaluator = LeaderboardEvaluator(arguments, statistics_manager) File "/home/demo/Public/LAV/leaderboard/leaderboard/leaderboard_evaluator.py", line 91, in init self.module_agent = importlib.import_module(module_name) File "/home/demo/anaconda3/envs/LAV-env/lib/python3.7/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 677, in _load_unlocked File "", line 728, in exec_module File "", line 219, in _call_with_frames_removed File "/home/demo/Public/LAV/team_code/lav_agent.py", line 15, in from models.lidar import LiDARModel ModuleNotFoundError: No module named 'models.lidar'

How can I fix this? Thanks.

Hi,

I wanted to know which scenario.json file was used for the ablations in this paper. I couldn't find the information in the README. The leaderboard version linked in the repository is from Jan 2021 it has one scenario file (all_towns_traffic_scenarios_public.json), but there seem to be only scenarios 1,3 and 4 specified within them. In the current master branch of the leaderboard there is also an all_towns_traffic_scenarios_public.json that additionally contains scenarios 2,7,8,9,10.

So, I am curious which scenario file was used for training and evaluation in this work (and which types of scenarios were considered)

Hello,

There is a problem that puzzles me. When running programs, the CPU looks strange. The number of running processes can reach 40! It makes the simulator no longer smooth and the system execution time longer.

I think my CPU can provide good support normally..

CPU: Intel® Xeon(R) CPU E5-2630 v4 @ 2.20GHz × 40 GPU: NVIDIA GeForce RTX 2080 T

May I ask if anyone has encountered a situation similar to mine？ Any help will be appreciated!

Hi, Chen. When I evaluate the method, it cost a lot of time. It took over 1 hour to finish the route_0 in routes_lav_valid.xml file while the time ratio was under 0.2. In addition, when I collecting data via Carla challange(LBC), I found the FPS is about 4 FPS that also quite time-consuming. And I found the CPU or memory of GPU are still have many available resources.

1. I want to ask the could I accelerate the process during evaluate or data collecting? Like by setting the FPS. Or the ratio only depend on my device performance.
2. To collect large dataset like LAV provided, did you start multiple processes to collect data simultaneously? How long did it took for collecting the provided dataset?

btw, I just use one TITAN Xp for above try. Looking forward to your reply.

Hello, Sorry to bother you.

In bev_planner_v2.py, when plotting the cropped_other_bev, I find that some bevs shift from their original their locs. As shown in Fig. (c), the locs are plotted as red dots, which shifts away from the vehicle colored with yellow. I have already set the feature_x_jitter, feature_y_jitter, x_jitter, a_jitter to 0. The crop_feature code can be found bellow. locs_jitter = (torch.rand((K,2))*2-1).float().to(locs.device) * self.feature_x_jitter locs_jitter[:,1] = 0 oris_jitter = (torch.rand((K,))*2-1).float().to(oris.device) * self.feature_angle_jitter cropped_other_bev = self.crop_feature(flat_bev, flat_rel_loc0+locs_jitter, flat_rel_ori0+oris_jitter, pixels_per_meter=self.pixels_per_meter, crop_size=self.crop_size*2)

How can I crop other bevs as Fig. (b)? Thank you for your time.

Hello, Thank you for your wonderful work.

When reading your code, I find that in temporal_lidar_painted_dataset.py locs[0] is equal to ego_locs. While, in temporal_bev_dataset.py, locs[0] is different from ego_locs. The disagreement caused by ego_locs = rotate_points(ego_locs, -angle, ego_locs[0]) + [offset/self.pixels_per_meter, 0] After rotating ego_locs, ego_locs[0] has already been changed, which will affect the following line. locs = rotate_points(locs, -angle, ego_locs[0]) + [offset/self.pixels_per_meter, 0] Is locs[0] supposed to be equivalent to ego_locs?

Thank you for your time.

Hi Chen. I find that the Semantic segmentation camera groups traffic lights together. But I'm not sure if I can distinguish between different colors of lights? Looking forward to your reply.

Hello,

I am a student working on a project involving testing ADAS/AV and scenario synthesis using BeamNG.tech and would love to run (test) your driving agent in that simulator. I know CARLA is kind of a de facto standard, but IMHO BeamNG.tech is superior when it comes to physic simulation, content, and flexibility. Further, BeamNG.tech is free for research, offers a python API, just like CARLA, and implements a wide range of sensors.

So I wonder how technically difficult it would be to port LAV to BeamNG.tech and whether anyone of you could support me (and my colleagues) in doing so. Hope to hear from you soon,

Thanks!

-- Benedikt Steininger

Thanks for your great work! It helped me a lot!

But I still have a question: When I run python data_collect.py --num-runners=8 in terminal, it gives me this error:

(pid=raylet)   File "/home/jianli/anaconda3/envs/LAV-env2/lib/python3.7/socket.py", line 752, in getaddrinfo
(pid=raylet)     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
(pid=raylet) socket.gaierror: [Errno -2] Name or service not known

Have anyone met this problem before? Is there anything I could do to solve this problem? Thank you in advance!

Hello @dotchen , impressive work! I have ran your latest LAV agent with CARLA Leaderboard. I try to understand the detailed implementation and have run across a couple of questions below:

1. The LAV agent plans 20 steps for the Ego vehicle. Are the 20 points in the ego_plan_locs distributed at equivalent time interval? If so, what is the time interval? And is the object prediction using the same time interval? According to the coordinates in the generated ego_plan_locs, a reasonable interval would be between 0.3s - 0.4s, assuming a same time interval for 20 steps.

2. In the new release, there's an EKF module. But I still observed quite some noise in the EKF output. The ego loc was still constantly jumping around in a small range. Is there any plan to improve this? Or am I missing something here?

I'd appreciate your insights on these questions a lot! Thanks in advance!

I found that in the previous V1 agent, the threshold for brake prediction is 0.3 in lav_agent.py

if float(pred_bra) > 0.3:
            throt, brake = 0, 1

However, the threshold changes to 0.1 in the V2 agent

if float(pred_bra) > 0.1:
            throt, brake = 0, 1

This makes the car stop in the middle road in many scenarios.

I personally change the code to use the throttle as threshold to fix the problem.

if float(pred_bra) > throt:
            throt, brake = 0, 1