Did Baidu really do better than Google’s Waymo in California’s self-driving report?
Chinese self-driving pioneer has a lower disengagement rate than the Google spinoff, but experts say the numbers don’t mean much
But industry experts say it’s not.
“The disengagement metric is essentially worthless and does not really tell anything about the performance of an automated driving system,” Sam Abuelsamid, an analyst at consultancy Navigant, told us. “While Baidu's results are indeed impressive this year, they really don't mean very much.”
For one, there is no standard for when a driver should disengage: Each company is free to decide on its own. The rate of disengagement also depends largely on where the vehicle was travelling.
“If the company does a lot of testing in areas with little traffic or other obstacles, there may be fewer disengagements than a company testing in a dense urban environment like San Francisco,” Abuelsamid explained.
Baidu’s self-driving arm, Apollo, agrees that using disengagement rates for comparison across companies is “not that meaningful”. It says the data is better used for examining a company’s performance over time. Baidu has gradually improved its testing scale and capability without any reported accidents, Apollo said.
China's robotaxis are high-tech but overly 'considerate'
Companies’ performances aren’t comparable across borders since testing criteria vary in different countries.
“In China, autonomous vehicle companies actually have to pass on-track performance evaluations before they can get a permit for on-road testing, something that does not happen in the US,” said Abuelsamid.
Even California’s DMV advises against treating its annual report as a scoreboard.
“[The reports] are not intended to compare one company with another or reach broad conclusions on technological capabilities,” said Marty Greenstein, a DMV spokesman. He added that companies aren’t required to report testing done on private roads, test tracks, out-of-state locations or virtual simulations.
Purchase the China AI Report 2020 brought to you by SCMP Research and enjoy a 20% discount (original price US$400). This 60-page all new intelligence report gives you first-hand insights and analysis into the latest industry developments and intelligence about China AI. Get exclusive access to our webinars for continuous learning, and interact with China AI executives in live Q&A. Offer valid until 31 March 2020.