Why we care about measuring autonomy performance…
I think the guys at Wayve.ai are unto something. They’re tackling the problem from a real-world approach. It’s practical, it’ll take time, but the successes, if indeed achievable will be huge.
The two fields, computer science, and mathematics, or even more accurately, the field of mathematics, have/has leveraged statistics as art to deduce and plan future trends and actions. Computer science with computer vision has been building programmetric models since at least the early 60’s with the use of computing language. Breakthroughs in performance and capability have started to see many new applications debut across different industries. The many contributing factors here are converging at defining some standardized benchmarks for use of AI and the deduction of independently operated and run systems.
The industry standard metric for measuring progress in autonomous driving is the California DMV’s “miles per intervention” metric. Many companies track internal scores by measuring interventions over fixed routes, which can be repeated daily. This is nowhere near sufficient, and many have ascribed their points of view around this. Wayve.ai has the following reasons:
- “Not all roads and cities are of equal complexity. An urban driving environment will naturally lead to a higher rate of intervention as opposed to an empty highway.
- Not all interventions are of equal severity. A slight adjustment in a trajectory through a turn may not have the same consequences as hitting the brakes to prevent a collision.
- A single high-level metric such as miles/intervention does not give us sufficient confidence for a safety argument, given the diversity and complexity of driving tasks.
- A single metric such as “number of interventions” does not give us enough feedback internally on where we should spend our time in order to improve the performance of the system.”
How is Wayve measuring and managing autonomy performance giving their advances in self-driving cars? Is scenarios-based evaluation enough?
Read more from Wayve at the following sources:
