Issues Regarding the Evaluation Results

[Louis-WD]

Thank you for open-sourcing the Bench2Drive work.
I ran the 220-route benchmark with our previous TCP reproduction and encountered the following problems:
When I inspect the local simulation recording, the vehicle drops straight down at the very first frame and keeps driving “in the void” for the rest of the episode.
The score_route is 100, yet score_penalty is 0. The failure message says:
“Agent went outside its route lanes for about 133.12 m” and
“min_speed_infractions: Average speed is 100 % of the surrounding traffic’s one.”
How should I interpret these numbers?
Could you help clarify these points?
Attached below are some simulation results

result.txt

[Louis-WD]

"index": 0,
"route_id": "RouteScenario_1711_rep0",
"scenario_name": "ParkingCutIn_1",
"weather_id": "15",
"save_name": "RouteScenario_1711_rep0_Town12_ParkingCutIn_1_15_10_21_10_11_13",
"status": "Completed",
"num_infractions": 23,
"infractions": {
"collisions_layout": [],
"collisions_pedestrian": [],
"collisions_vehicle": [],
"red_light": [],
"stop_infraction": [],
"outside_route_lanes": [
"Agent went outside its route lanes for about 133.138 meters (100.0% of the completed route)"
],
"min_speed_infractions": [
"Average speed is 0.06% of the surrounding traffic's one",
"Average speed is 100% of the surrounding traffic's one",
"Average speed is 100% of the surrounding traffic's one",
"Average speed is 100% of the surrounding traffic's one",
"Average speed is 100% of the surrounding traffic's one",
"Average speed is 100% of the surrounding traffic's one",
"Average speed is 100% of the surrounding traffic's one",
"Average speed is 100% of the surrounding traffic's one",
"Average speed is 100% of the surrounding traffic's one",
"Average speed is 100% of the surrounding traffic's one",
"Average speed is 100% of the surrounding traffic's one",
"Average speed is 100% of the surrounding traffic's one",
"Average speed is 100% of the surrounding traffic's one",
"Average speed is 100% of the surrounding traffic's one",
"Average speed is 100% of the surrounding traffic's one",
"Average speed is 100% of the surrounding traffic's one",
"Average speed is 100% of the surrounding traffic's one",
"Average speed is 100% of the surrounding traffic's one",
"Average speed is 100% of the surrounding traffic's one",
"Average speed is 100% of the surrounding traffic's one",
"Average speed is 100% of the surrounding traffic's one"
],
"yield_emergency_vehicle_infractions": [],
"scenario_timeouts": [],
"route_dev": [
"Agent deviated from the route at (x=-497.5, y=3702.965, z=-37671.559)"
],
"vehicle_blocked": [],
"route_timeout": []
},
"scores": {
"score_route": 100.0,
"score_penalty": 0.0,
"score_composed": 0.0
},
"meta": {
"route_length": 133.138,
"duration_game": 87.3,
"duration_system": 134.152
},
"town_name": "Town12"

"index": 1,
"route_id": "RouteScenario_1773_rep0",
"scenario_name": "ParkedObstacle_1",
"weather_id": "25",
"save_name": "RouteScenario_1773_rep0_Town12_ParkedObstacle_1_25_10_21_10_14_02",
"status": "Failed - TickRuntime",
"num_infractions": 0,
"infractions": {
"collisions_layout": [],
"collisions_pedestrian": [],
"collisions_vehicle": [],
"red_light": [],
"stop_infraction": [],
"outside_route_lanes": [],
"min_speed_infractions": [],
"yield_emergency_vehicle_infractions": [],
"scenario_timeouts": [],
"route_dev": [],
"vehicle_blocked": [],
"route_timeout": []
},
"scores": {
"score_route": 0,
"score_penalty": 1.0,
"score_composed": 0.0
},
"meta": {
"route_length": 132.062,
"duration_game": 200.05,
"duration_system": 229.345
},
"town_name": "Town12"