Make sure you have read FAQ before posting. Thanks!

When I tried to use pretrained model to origin leaderboard, the image_agent.py sensors config is not correct with WOR team submit on leaderboard benchmark

And for the image_agent posted on this repo:

1. the collision and stitch_camera is not allowed in origin leaderboard
2. the width and height is fixed by origin leaderboard

        sensors = [
            {'type': 'sensor.collision', 'id': 'COLLISION'},
            {'type': 'sensor.speedometer', 'id': 'EGO'},
            {'type': 'sensor.other.gnss', 'x': 0., 'y': 0.0, 'z': self.camera_z, 'id': 'GPS'},
            {'type': 'sensor.stitch_camera.rgb', 'x': self.camera_x, 'y': 0, 'z': self.camera_z, 'roll': 0.0, 'pitch': 0.0, 'yaw': 0.0,
            'width': 160, 'height': 240, 'fov': 60, 'id': f'Wide_RGB'},
            {'type': 'sensor.camera.rgb', 'x': self.camera_x, 'y': 0, 'z': self.camera_z, 'roll': 0.0, 'pitch': 0.0, 'yaw': 0.0,
            'width': 384, 'height': 240, 'fov': 50, 'id': f'Narrow_RGB'},
        ]

I want to see from the origin leaderboard because I want to know about the failure situation on WOR and LBC since the result analysis about fail is lack.

If it's possible could you pls give the config/agent and pretrained model files submit to leaderboard (like using another branch for origin leaderboard) or upload the result files after submiting leaderboard , either one is ok for analysis fail situation.

i run carla and worldinrails in server ; ubuntu 184, aioredis==1.3.0

run carla: DISPLAY= ${CARLA_ROOT}/CarlaUE4.sh -world-port=$port -opengl -quality-level=low -world-port=$port -resx=400 -resy=300 $ change in file scripts/launch_carla.sh

pygame no screen change settint : +++ b/leaderboard/leaderboard/leaderboard_evaluator.py @@ -23,7 +23,7 @@ import pkg_resources import sys import carla import signal

+os.environ['SDL_VIDEODRIVER'] = 'dummy'

when run: python -m rails.data_phase0 --num-runners=[NUM RUNNERS] --port=[WORLD PORT]
program run and exit no outdata and no error

when rails/data_phase0.py change this line ray.init(logging_level=40, local_mode=True, log_to_driver=False)

local_mode=True from default False to True

it run ok;

what happen?

Hey Dian,

Trying to run data_phase2 and I get the following Ray error (seems to have issue with RemoteMainDataset constructor?). I did some debugging by replacing all the @ray.remote stuff and .remote() commands with the non-ray versions, and the code runs with no issue (although the progress bar didn't progress past 0 frames after a minute or two, not quite sure if it's supposed to take that long or not).

Did you ever see anything like this/know what I should do?

(wor) aaron@Aarons-Machine:~/workspace/carla/WorldOnRails$ RAY_PDB=1 python -m rails.data_phase2 --num-workers=12
Traceback (most recent call last):
  File "/home/aaron/anaconda3/envs/wor/lib/python3.7/runpy.py", line 193, in _run_module_as_main
2021-05-29 14:45:49,862 WARNING worker.py:1034 -- Traceback (most recent call last):
  File "/home/aaron/anaconda3/envs/wor/lib/python3.7/site-packages/ray/function_manager.py", line 251, in get_execution_info
    info = self._function_execution_info[job_id][function_id]
KeyError: FunctionID(41f68a98bcf1c9ebc84e01b0819040089631493c)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "python/ray/_raylet.pyx", line 550, in ray._raylet.task_execution_handler
  File "python/ray/_raylet.pyx", line 364, in ray._raylet.execute_task
  File "/home/aaron/anaconda3/envs/wor/lib/python3.7/site-packages/ray/function_manager.py", line 256, in get_execution_info
    raise KeyError(message)
KeyError: 'Error occurs in get_execution_info: job_id: JobID(01000000), function_descriptor: {type=PythonFunctionDescriptor, module_name=rails.datasets.main_dataset, class_name=RemoteMainDataset, function_name=__init__, function_hash=084f10af-7af1-46d7-8dda-ada171c2aad9}. Message: FunctionID(41f68a98bcf1c9ebc84e01b0819040089631493c)'
An unexpected internal error occurred while the worker was executing a task.
    "__main__", mod_spec)
  File "/home/aaron/anaconda3/envs/wor/lib/python3.7/runpy.py", line 85, in _run_code
2021-05-29 14:45:49,862 WARNING worker.py:1034 -- A worker died or was killed while executing task ffffffffffffffffcb230a5701000000.
    exec(code, run_globals)
  File "/home/aaron/workspace/carla/WorldOnRails/rails/data_phase2.py", line 67, in <module>
    main(args)
  File "/home/aaron/workspace/carla/WorldOnRails/rails/data_phase2.py", line 13, in main
    total_frames = ray.get(dataset.num_frames.remote())
  File "/home/aaron/anaconda3/envs/wor/lib/python3.7/site-packages/ray/worker.py", line 1381, in get
    raise value
ray.exceptions.RayActorError: The actor died unexpectedly before finishing this task.
(wor) aaron@Aarons-Machine:~/workspace/carla/WorldOnRails$

Make sure you have read FAQ before posting. Thanks! Hi, I would like to know How to set stop conditions of LBC agent during collecting data .By reading the code, I found that the 'q_collecter' and 'random_collecter' will stop collecting and save the data until the collision. Now, I want to try to train this version of LBC agent. I think the termination condition for LBC agent to collect data is arriving the destination.But I don't know how to implement it. Can you give me some suggestions or share this part of your code . Thank you , and I will be looking forward to your reply.

Hey again, just wanted to ask for a bit more training detail. I'm trying to reproduce results (computing benchmark metrics on the Leaderboard validation routes), and the pretrained weights seem to do better than the weights I train on my own setup. I have two questions about how main_model_10.th is trained

1. How large was the main_dataset used to train that model?
2. How long did it take to train the model? I'm assuming it was 10 epochs total, which would've taken multiple days for the 100 Gb main dataset I was using

Also, thank you for the reply on the other issue - I'll need to mull over your answers and read through the code some more to understand what's going on. I'll reply to that issue once I've thought about it some more!

Make sure you have read FAQ before posting. Thanks! Hello, After running all the above programs correctly as you suggested，I have trained the ego-model and successfully collected the nocrash data about 186GB, then what I need to do is label Q. So I run with $python -m rails.data_phase2 --num-workers=4, it shows as follows:

|         | 0/53267 [00:00<?, ?it/s

And I have checked my GPU, and it shows the ray and running now.

+-------------------------------+----------------------+----------------------+
|   2  GeForce RTX 2080    Off  | 00000000:86:00.0 Off |                  N/A |
| 50%   71C    P2   153W / 215W |   2480MiB /  7952MiB |     98%      Default |
+-------------------------------+----------------------+----------------------+
|   3  GeForce RTX 2080    Off  | 00000000:AF:00.0 Off |                  N/A |
| 51%   72C    P2   152W / 215W |   2480MiB /  7952MiB |     98%      Default |

|    2     14845      C   ray::RAILSActionLabeler.run()               1229MiB |
|    2     14855      C   ray::RAILSActionLabeler.run()               1229MiB |
|    3     14898      C   ray::RAILSActionLabeler.run()               1229MiB |
|    3     14919      C   ray::RAILSActionLabeler.run()               1229MiB 

When I tried CTRL+C, it shows:

raceback (most recent call last):
  File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/shy/Desktop/WorldOnRails/rails/data_phase2.py", line 58, in <module>
    main(args)
  File "/home/shy/Desktop/WorldOnRails/rails/data_phase2.py", line 24, in main
    current_frames = ray.get(logger.total_frames.remote())
  File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/ray/worker.py", line 1372, in get
    object_refs, timeout=timeout)
  File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/ray/worker.py", line 304, in get_objects
    object_refs, self.current_task_id, timeout_ms)
  File "python/ray/_raylet.pyx", line 869, in ray._raylet.CoreWorker.get_objects
  File "python/ray/_raylet.pyx", line 142, in ray._raylet.check_status
KeyboardInterrupt
^CError in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/ray/node.py", line 868, in _kill_process_type
    process.wait(timeout_seconds)
  File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/subprocess.py", line 1019, in wait
    return self._wait(timeout=timeout)
  File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/subprocess.py", line 1647, in _wait
    time.sleep(delay)
KeyboardInterrupt
  0%|                              

It seems it just waiting now? (But we donot need to launch carla in this phase) And the data-dir is set to the collected data direction, the config.yaml is set to the no-crash config. ( default='/home/shy/Desktop/WorldOnRails/experiments/config_nocrash.yaml')), just cp config.yaml to experiments file. Thanks a lot!

Make sure you have read FAQ before posting. Thanks! Hello， Sorry to disrupt you again. After reading the FAQ carefully, I downloaded your released datasets, there are some small mistakes here. And there is some problem about launching carla, I guess this problem leads to the errors about training the ego model.

1 datasets lose key and value

For example,

real_file='/home/shy/Desktop/aazijolwvf'
json_file=real_file+'/data.json'
data = json.load(open(json_file))
data['0']
{'loc': [655.82763671875, 30.739931106567383, 0.03367812931537628],
 'rot': [-91.5350341796875],
 'spd': [2.312199831008911],
 'cmd': [4.0],
 'lbl_00': '/ssd2/dian/challenge_data/main_trajs6_converted/aazijolwvf/rgbs/lbl_00_00000.png',
 'lbl_01': '/ssd2/dian/challenge_data/main_trajs6_converted/aazijolwvf/rgbs/lbl_01_00000.png',
 'lbl_02': '/ssd2/dian/challenge_data/main_trajs6_converted/aazijolwvf/rgbs/lbl_02_00000.png',
 'lbl_03': '/ssd2/dian/challenge_data/main_trajs6_converted/aazijolwvf/rgbs/lbl_03_00000.png',
 'lbl_04': '/ssd2/dian/challenge_data/main_trajs6_converted/aazijolwvf/rgbs/lbl_04_00000.png',
 'lbl_05': '/ssd2/dian/challenge_data/main_trajs6_converted/aazijolwvf/rgbs/lbl_05_00000.png',
 'lbl_06': '/ssd2/dian/challenge_data/main_trajs6_converted/aazijolwvf/rgbs/lbl_06_00000.png',
 'lbl_07': '/ssd2/dian/challenge_data/main_trajs6_converted/aazijolwvf/rgbs/lbl_07_00000.png',
 'lbl_08': '/ssd2/dian/challenge_data/main_trajs6_converted/aazijolwvf/rgbs/lbl_08_00000.png',
 'lbl_09': '/ssd2/dian/challenge_data/main_trajs6_converted/aazijolwvf/rgbs/lbl_09_00000.png',
 'lbl_10': '/ssd2/dian/challenge_data/main_trajs6_converted/aazijolwvf/rgbs/lbl_10_00000.png',
 'lbl_11': '/ssd2/dian/challenge_data/main_trajs6_converted/aazijolwvf/rgbs/lbl_11_00000.png',
 'wide_rgb_{}': '/rgbs/narr_rgb_2_00000.jpg',
 'wide_sem_{}': '/rgbs/wide_sem_2_00000.png',
 'narr_sem_{}': '/rgbs/narr_sem_2_00000.png'}

It seems that the narr_rgb_{} key and value is lost, and the file name does not correspond to the value in this json file. Besides, the value of wide_rgb_{} is also needs to be changed. So the code here is provided:

import json
real_file='/home/shy/Desktop/aazijolwvf'
#json_file ='/home/shy/Desktop/aazijolwvf/data.json'
json_file=real_file+'/data.json'
json_save_file=real_file+'/data_new.json'
data = json.load(open(json_file))

for i in range(data['len']):
    idx = str(i)
    data[idx]['narr_rgb_{}']=data[idx]['wide_rgb_{}']
    wide_value = data[idx]['wide_rgb_{}'][:6]+'wide'+data[idx]['wide_rgb_{}'][10:]
    #print("wide value is", wide_value)
    data[idx]['wide_rgb_{}']=wide_value

    # data['0'].keys()
    key_change_lbl = [key for key in data[idx].keys()][4:16]
    key_change_rgb=[ [key for key in data[idx].keys()][16],[key for key in data[idx].keys()][19] ]
    key_change_sem=[ [key for key in data[idx].keys()][17],[key for key in data[idx].keys()][18] ]
    for key in key_change_lbl:
        #print("data",data['0'][key])
        real_str = data[idx][key][58:]
        data_new=real_file+real_str
        data[idx][key] = data_new
    for key in key_change_rgb:
    # remove_rgb
        real_str_first= data[idx][key][:10]
        real_str_last= data[idx][key][14:]
        data_new=real_file+real_str_first+real_str_last
        data[idx][key] = data_new

    for key in key_change_sem:
        real_str= data[idx][key]
        data_new=real_file+real_str
        data[idx][key] = data_new
    

with open(json_save_file, 'w') as outfile:
    json.dump(data, outfile)

And for testing, we set a random number, the code is following:

import numpy as np
idx = np.random.randint(0,200)
print("The index is",idx)
idx = str(idx)
key_images= [key for key in data[idx].keys()][4:]
for key in key_images[12:]:
    print(data[idx][key])
    x = plt.imread(data[idx][key])
   # print("x.shape",x.shape)
    plt.figure()
    plt.imshow(x)

And finally the datasets can be loaded.

2 what does 6+:lane centers mean?

I found 6-11 are lane centers,what is the difference between them or how to collect it? I thought the direction of no.11 images seems different, I guess it maybe the people's lane center?

3 Is Vulcan necessary for Carla?

I failed to launch Carla by running the launch Carla shell. In FAQ, it might be the error of Vulcan, However, I can still launch Carla using the OpenGL. And I successfully test the examples in PythonAPI. After launching Carla, I decided to train the ego_model. And first of all, I need to collect the datasets.

$ ./CarlaUE4.sh -fps 10 -world-port 2000

Then I tried

$ python -m rails.data_phase0 --num-runners=1 --port=2000
(world_on_rails) shy@carla:~/WorldOnRails0526$ python -m rails.data_phase0 --num-runners=1 --port=2000
pygame 2.0.1 (SDL 2.0.14, Python 3.7.9)
Hello from the pygame community. https://www.pygame.org/contribute.html
Traceback (most recent call last):
  File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/shy/WorldOnRails0526/rails/data_phase0.py", line 53, in <module>
    main(args)
  File "/home/shy/WorldOnRails0526/rails/data_phase0.py", line 16, in main
    runner = ScenarioRunner.remote(args, scenario, route, port=port, tm_port=tm_port)
  File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/ray/actor.py", line 407, in remote
    return self._remote(args=args, kwargs=kwargs)
  File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/ray/actor.py", line 658, in _remote
    kwargs)
  File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/ray/signature.py", line 116, in flatten_args
    raise TypeError(str(exc)) from None
TypeError: missing a required argument: 'route'

I am not sure how to fix this problem, I just do as follows:（Did I set the right scenario_class?）

runner = ScenarioRunner.remote(args,scenario_class='route_scenario', scenario=scenario, route=route, port=port, tm_port=tm_port)

After that, there is no error and it seems that the code is running at ray.wait(jobs, num_returns=args.num_runners), and no data are saved. Then I turned to phase1,

python -m rails.data_phase1 --scenario=train_scenario --num-runners=1 --port 2000

(world_on_rails) shy@carla:~/WorldOnRails0526$ python -m rails.data_phase1 --scenario=train_scenario --num-runners=1 --port 2000
pygame 2.0.1 (SDL 2.0.14, Python 3.7.9)
Hello from the pygame community. https://www.pygame.org/contribute.html
(pid=15754) pygame 2.0.1 (SDL 2.0.14, Python 3.7.9)
(pid=15754) Hello from the pygame community. https://www.pygame.org/contribute.html
(pid=raylet) Traceback (most recent call last):
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/ray/new_dashboard/agent.py", line 317, in <module>
(pid=raylet)     raise e
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/ray/new_dashboard/agent.py", line 306, in <module>
(pid=raylet)     loop.run_until_complete(agent.run())
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
(pid=raylet)     return future.result()
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/ray/new_dashboard/agent.py", line 180, in run
(pid=raylet)     agent_ip_address=self.ip))
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/grpc/aio/_call.py", line 286, in __await__
(pid=raylet)     self._cython_call._status)
(pid=raylet) grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
(pid=raylet) 	status = StatusCode.UNAVAILABLE
(pid=raylet) 	details = "DNS resolution failed for service: https:"
(pid=raylet) 	debug_error_string = "{"created":"@1622118924.664088343","description":"Resolver transient failure","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":2132,"referenced_errors":[{"created":"@1622118924.664086974","description":"DNS resolution failed for service: https:","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/dns_resolver_ares.cc","file_line":361,"grpc_status":14,"referenced_errors":[{"created":"@1622118924.664067929","description":"C-ares status is not ARES_SUCCESS qtype=A name=https is_balancer=0: Could not contact DNS servers","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc","file_line":716,"referenced_errors":[{"created":"@1622118924.663999265","description":"C-ares status is not ARES_SUCCESS qtype=AAAA name=https is_balancer=0: Could not contact DNS servers","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc","file_line":716}]}]}]}"
(pid=raylet) >
(pid=raylet) Traceback (most recent call last):
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/ray/new_dashboard/agent.py", line 317, in <module>
(pid=raylet)     raise e
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/ray/new_dashboard/agent.py", line 306, in <module>
(pid=raylet)     loop.run_until_complete(agent.run())
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
(pid=raylet)     return future.result()
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/ray/new_dashboard/agent.py", line 180, in run
(pid=raylet)     agent_ip_address=self.ip))
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/grpc/aio/_call.py", line 286, in __await__
(pid=raylet)     self._cython_call._status)
(pid=raylet) grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
(pid=raylet) 	status = StatusCode.UNAVAILABLE
(pid=raylet) 	details = "DNS resolution failed for service: https:"
(pid=raylet) 	debug_error_string = "{"created":"@1622118926.269346249","description":"Resolver transient failure","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":2132,"referenced_errors":[{"created":"@1622118926.269345149","description":"DNS resolution failed for service: https:","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/dns_resolver_ares.cc","file_line":361,"grpc_status":14,"referenced_errors":[{"created":"@1622118926.269326571","description":"C-ares status is not ARES_SUCCESS qtype=A name=https is_balancer=0: Could not contact DNS servers","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc","file_line":716,"referenced_errors":[{"created":"@1622118926.269311059","description":"C-ares status is not ARES_SUCCESS qtype=AAAA name=https is_balancer=0: Could not contact DNS servers","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc","file_line":716}]}]}]}"
(pid=raylet) >
(pid=raylet) Traceback (most recent call last):
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/ray/new_dashboard/agent.py", line 317, in <module>
(pid=raylet)     raise e
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/ray/new_dashboard/agent.py", line 306, in <module>
(pid=raylet)     loop.run_until_complete(agent.run())
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
(pid=raylet)     return future.result()
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/ray/new_dashboard/agent.py", line 180, in run
(pid=raylet)     agent_ip_address=self.ip))
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/grpc/aio/_call.py", line 286, in __await__
(pid=raylet)     self._cython_call._status)
(pid=raylet) grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
(pid=raylet) 	status = StatusCode.UNAVAILABLE
(pid=raylet) 	details = "DNS resolution failed for service: https:"
(pid=raylet) 	debug_error_string = "{"created":"@1622118927.908004098","description":"Resolver transient failure","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":2132,"referenced_errors":[{"created":"@1622118927.908003298","description":"DNS resolution failed for service: https:","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/dns_resolver_ares.cc","file_line":361,"grpc_status":14,"referenced_errors":[{"created":"@1622118927.907989921","description":"C-ares status is not ARES_SUCCESS qtype=A name=https is_balancer=0: Could not contact DNS servers","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc","file_line":716,"referenced_errors":[{"created":"@1622118927.907964504","description":"C-ares status is not ARES_SUCCESS qtype=AAAA name=https is_balancer=0: Could not contact DNS servers","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc","file_line":716}]}]}]}"
(pid=raylet) >
(pid=raylet) Traceback (most recent call last):
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/ray/new_dashboard/agent.py", line 317, in <module>
(pid=raylet)     raise e
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/ray/new_dashboard/agent.py", line 306, in <module>
(pid=raylet)     loop.run_until_complete(agent.run())
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
(pid=raylet)     return future.result()
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/ray/new_dashboard/agent.py", line 180, in run
(pid=raylet)     agent_ip_address=self.ip))
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/grpc/aio/_call.py", line 286, in __await__
(pid=raylet)     self._cython_call._status)
(pid=raylet) grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
(pid=raylet) 	status = StatusCode.UNAVAILABLE
(pid=raylet) 	details = "DNS resolution failed for service: https:"
(pid=raylet) 	debug_error_string = "{"created":"@1622118929.553582429","description":"Resolver transient failure","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":2132,"referenced_errors":[{"created":"@1622118929.553581561","description":"DNS resolution failed for service: https:","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/dns_resolver_ares.cc","file_line":361,"grpc_status":14,"referenced_errors":[{"created":"@1622118929.553567170","description":"C-ares status is not ARES_SUCCESS qtype=A name=https is_balancer=0: Could not contact DNS servers","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc","file_line":716,"referenced_errors":[{"created":"@1622118929.553544142","description":"C-ares status is not ARES_SUCCESS qtype=AAAA name=https is_balancer=0: Could not contact DNS servers","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc","file_line":716}]}]}]}"
(pid=raylet) >
(pid=raylet) Traceback (most recent call last):
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/ray/new_dashboard/agent.py", line 317, in <module>
(pid=raylet)     raise e
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/ray/new_dashboard/agent.py", line 306, in <module>
(pid=raylet)     loop.run_until_complete(agent.run())
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
(pid=raylet)     return future.result()
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/ray/new_dashboard/agent.py", line 180, in run
(pid=raylet)     agent_ip_address=self.ip))
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/grpc/aio/_call.py", line 286, in __await__
(pid=raylet)     self._cython_call._status)
(pid=raylet) grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
(pid=raylet) 	status = StatusCode.UNAVAILABLE
(pid=raylet) 	details = "DNS resolution failed for service: https:"
(pid=raylet) 	debug_error_string = "{"created":"@1622118931.222429438","description":"Resolver transient failure","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":2132,"referenced_errors":[{"created":"@1622118931.222428625","description":"DNS resolution failed for service: https:","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/dns_resolver_ares.cc","file_line":361,"grpc_status":14,"referenced_errors":[{"created":"@1622118931.222411772","description":"C-ares status is not ARES_SUCCESS qtype=A name=https is_balancer=0: Could not contact DNS servers","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc","file_line":716,"referenced_errors":[{"created":"@1622118931.222371125","description":"C-ares status is not ARES_SUCCESS qtype=AAAA name=https is_balancer=0: Could not contact DNS servers","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc","file_line":716}]}]}]}"
(pid=raylet) >
(pid=raylet) Traceback (most recent call last):
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/ray/new_dashboard/agent.py", line 317, in <module>
(pid=raylet)     raise e
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/ray/new_dashboard/agent.py", line 306, in <module>
(pid=raylet)     loop.run_until_complete(agent.run())
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
(pid=raylet)     return future.result()
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/ray/new_dashboard/agent.py", line 180, in run
(pid=raylet)     agent_ip_address=self.ip))
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/grpc/aio/_call.py", line 286, in __await__
(pid=raylet)     self._cython_call._status)
(pid=raylet) grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
(pid=raylet) 	status = StatusCode.UNAVAILABLE
(pid=raylet) 	details = "DNS resolution failed for service: https:"
(pid=raylet) 	debug_error_string = "{"created":"@1622118932.839846720","description":"Resolver transient failure","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":2132,"referenced_errors":[{"created":"@1622118932.839845839","description":"DNS resolution failed for service: https:","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/dns_resolver_ares.cc","file_line":361,"grpc_status":14,"referenced_errors":[{"created":"@1622118932.839830135","description":"C-ares status is not ARES_SUCCESS qtype=A name=https is_balancer=0: Could not contact DNS servers","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc","file_line":716,"referenced_errors":[{"created":"@1622118932.839817847","description":"C-ares status is not ARES_SUCCESS qtype=AAAA name=https is_balancer=0: Could not contact DNS servers","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc","file_line":716}]}]}]}"
(pid=raylet) >
(pid=raylet) Traceback (most recent call last):
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/ray/new_dashboard/agent.py", line 317, in <module>
(pid=raylet)     raise e
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/ray/new_dashboard/agent.py", line 306, in <module>
(pid=raylet)     loop.run_until_complete(agent.run())
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
(pid=raylet)     return future.result()
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/ray/new_dashboard/agent.py", line 180, in run
(pid=raylet)     agent_ip_address=self.ip))
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/grpc/aio/_call.py", line 286, in __await__
(pid=raylet)     self._cython_call._status)
(pid=raylet) grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
(pid=raylet) 	status = StatusCode.UNAVAILABLE
(pid=raylet) 	details = "DNS resolution failed for service: https:"
(pid=raylet) 	debug_error_string = "{"created":"@1622118934.437792694","description":"Resolver transient failure","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":2132,"referenced_errors":[{"created":"@1622118934.437791878","description":"DNS resolution failed for service: https:","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/dns_resolver_ares.cc","file_line":361,"grpc_status":14,"referenced_errors":[{"created":"@1622118934.437777832","description":"C-ares status is not ARES_SUCCESS qtype=A name=https is_balancer=0: Could not contact DNS servers","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc","file_line":716,"referenced_errors":[{"created":"@1622118934.437756242","description":"C-ares status is not ARES_SUCCESS qtype=AAAA name=https is_balancer=0: Could not contact DNS servers","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc","file_line":716}]}]}]}"
(pid=raylet) >
(pid=raylet) Traceback (most recent call last):
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/ray/new_dashboard/agent.py", line 317, in <module>
(pid=raylet)     raise e
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/ray/new_dashboard/agent.py", line 306, in <module>
(pid=raylet)     loop.run_until_complete(agent.run())
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
(pid=raylet)     return future.result()
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/ray/new_dashboard/agent.py", line 180, in run
(pid=raylet)     agent_ip_address=self.ip))
(pid=raylet)   File "/home/shy/anaconda3/envs/world_on_rails/lib/python3.7/site-packages/grpc/aio/_call.py", line 286, in __await__
(pid=raylet)     self._cython_call._status)
(pid=raylet) grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
(pid=raylet) 	status = StatusCode.UNAVAILABLE
(pid=raylet) 	details = "DNS resolution failed for service: https:"
(pid=raylet) 	debug_error_string = "{"created":"@1622118936.032766161","description":"Resolver transient failure","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":2132,"referenced_errors":[{"created":"@1622118936.032765376","description":"DNS resolution failed for service: https:","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/dns_resolver_ares.cc","file_line":361,"grpc_status":14,"referenced_errors":[{"created":"@1622118936.032752221","description":"C-ares status is not ARES_SUCCESS qtype=A name=https is_balancer=0: Could not contact DNS servers","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc","file_line":716,"referenced_errors":[{"created":"@1622118936.032727375","description":"C-ares status is not ARES_SUCCESS qtype=AAAA name=https is_balancer=0: Could not contact DNS servers","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc","file_line":716}]}]}]}"
(pid=raylet) >
2021-05-27 20:35:37,323	ERROR worker.py:980 -- Possible unhandled error from worker: ray::ScenarioRunner.run() (pid=15754, ip=219.216.98.5)
  File "python/ray/_raylet.pyx", line 463, in ray._raylet.execute_task
  File "python/ray/_raylet.pyx", line 415, in ray._raylet.execute_task.function_executor
  File "/home/shy/WorldOnRails0526/runners/scenario_runner.py", line 27, in run
    return self.runner.run(self.args)
  File "/home/shy/WorldOnRails0526/leaderboard/leaderboard/leaderboard_evaluator.py", line 417, in run
    self.statistics_manager.clear_record(args.checkpoint)
  File "/home/shy/WorldOnRails0526/leaderboard/leaderboard/utils/statistics_manager.py", line 355, in clear_record
    with open(endpoint, 'w') as fd:
FileNotFoundError: [Errno 2] No such file or directory: 'results/00_simulation_results.json'

The same error can be seen in the phase2. I guess these errors are caused by the ray.Now I am confused by these data collecting errors. And I would appreciate it if you could help me fixed these errors. Thank you again!

Hi,

Thank you providing the code. I was trying to reproduce the results on the noCrash benchmark from the paper and my reproduced results (Column 4) are as follows in comparison to the results given by your implementation (Column 3). I am using the pre-trained model for the noCrash benchmark.

| Town, weather | Traffic | RAILS Result (from paper) | Reproduced result | | ------------- | ------------- | ------------- | ------------- | | Town 01, Train Weather | Empty | 98 | 100 | | | Regular | 100 | 95 | | | Dense | 96 | 82 | | Town 01, Test Weather | Empty | 90 | 80 | | | Regular | 90 | 39 | | | Dense | 84 | 34 | | Town 02, Train Weather | Empty | 94 | 78 | | | Regular | 89 | 63 | | | Dense | 74 | 46 | Town 02, Test Weather | Empty | 78 | 36 | | | Regular | 82 | 34 | | | Dense | 66 | 24 |

Can you please let me know what can be the issue? I followed the installation instructions and I am using the evaluate_nocrash.py command under the noCrash routes.

Any help would be greatly appreciated.

Thanks, Himangi

Hello, I am very interested in your project. The new version of Carla enriches many functions and fixes some bugs. The project is great. If the project is compatible with the latest version of Carla, it will have more extensions. Will it be possible to release the training code or model under carla0.9.12+ ? Or I just wonder if the code will also work in carla0.9.12+ or is it relatively easy to extend it for that?

Thanks, in advance!

Hello! I want to change the camera I get when I run evaluate_nocrash.py from bird view to a dashboard camera.
Things I tried to do: -I tried to modify the code so I got the view from a dashboard sensor but I couldn't exactly pinpoint what I needed to change and where. -I thought about recording the session and then viewing it by the sensor I wanted, but the --record argument is only available to the nocrash_evaluator.py and I am having a hard time running it as well. So can you guide me to the right direction?

Raising a new issue:

While running data_phase1 in parallel, I often get the following error, always when trying to turn traffic_manager's sync mode on or off.

File "/home/aaronhua/WorldOnRails/leaderboard/leaderboard/leaderboard_evaluator.py", line 162, in _cleanup self.traffic_manager.set_synchronous_mode(False) RuntimeError: rpc::timeout: Timeout of 2000ms while calling RPC function 'set_synchronous_mode'

(pid=55313) File "/home/aaronhua/WorldOnRails/leaderboard/leaderboard/leaderboard_evaluator.py", line 238, in _load_and_wait_for_world (pid=55313) self.traffic_manager.set_synchronous_mode(True) (pid=55313) RuntimeError: rpc::timeout: Timeout of 2000ms while calling RPC function 'set_synchronous_mode'

My guess is that when running in parallel, if one worker happens to be writing data to disk at the end of a run while another worker is trying to reset the simulator, the entire system slows down and triggers the 2 second traffic_manager timeout. I modified the scripts/launch_carla to default to a server timeout of like 20 minutes but still get the above 2s timeout message. I've tried looking around to see if we can specify the traffic_manager specific timeout, but there's no comprehensive list of CarlaUE4.sh command line options.

The closest I can find is here, implying that the C++ implementation of the TrafficManager has a method that sets the set_synchronous_mode timeout. The Python bindings don't appear to have this method though. Do you have an idea of how I'd resolve this issue?

EDIT: as for your comment in the old thread, I've tried looking for errors that precede this one but haven't been able to find any. The only thing is ERROR: failed to destroy actor 11571 : unable to destroy actor: not found and the like, which appears when cleaning up a completed run and doesn't appear to actually break anything. I thought it was a timeout issue because I'm usually able to progress through a few routes and repetitions and gather valid data before workers will start to fail because of the error.

Hi Chen. Thanks for a lot of open source works and I am trying to follow them. I jump from the data collecting about LAV as you said there are similarities between them. I found that the random collector return a random control including throttle and steer. I feel confused how could the agent arrived the goal under the random control? ~~Does WOR and LAV use the same agent?~~ How does it chose random routes?

I am not able to find where are the waypoint rewards smoothly penalized based on the distance as written in the paper. I see that the orientation and speeds are interpolated while calculating the waypoint rewards but I do not see any processing related to penalizing based on the distance down to a value of 0.

It will be very helpful if somebody can point me in the right direction. Thanks

First of all, thank you for publishing this work. I would like to only retrain the Q table again by manipulating the reward function. So, I download the 100GB dataset and ran the data_phase2 module. However, because the dataset that you have provided doesn't have any *.mdb files, lmdb produces an error here because the read-only flag is true and says the no such file or directory. The path that I set is the root of this 1M record dataset.

        # Load dataset
        for full_path in glob.glob(f'{data_dir}/**'):
            txn = lmdb.open(
                full_path,
                max_readers=1, readonly=True,
                lock=False, readahead=False, meminit=False).begin(write=False)

Is there anything that I am doing wrong? Thanks.

Hi, may I know what does the thickness of the lines in lane images in the asset folder mean? For example in Town03, I cannot seem to understand why there are some lines that are thicker than the other.

Hello, I want to know more about the map. First, does the channel 'stop' represent the location information of traffic lights ? Second, What information does the 6 channels of 'waypoints' represent ？ Thank you！

Hello! Thanks for your working, but I have some questions:

1. How to control the number of cars and pedestrians?
2. How to create my own route instead of training or testing route?