Autonomous Autos 2018 – Rotten Apple

Waymo’s lead shrinks again to 2 to 1. Apple’s offering is awful.

• Analysis of autonomous testing data submitted to the California DMV for 2018, reveals that while Waymo is still out front, GM Cruise has made up ground, cementing its No.2 position.
• Newcomer Apple came in 2^nd last only to Uber whose performance has deteriorated massively in the last 12 months and was the only player to report more disengagements than miles driven.
• The main observations are:
  • First, experience still pays: There is still a strong correlation between miles driven and quality of offering.
  • The raw R² is 96% but I suspect that this is massively skewed by the two leaders who have driven 5.5x more miles than everyone else put together.
  • This continues to support RFM’s view that excellence in AI is more about experience than quantities or raw data on hand.
  • The outlier here is Apple who drove 78,605 miles and suffered 69,680 disengagements; nearly 1 for every mile driven and some 9,787x worse than Waymo.
  • The data clearly shows that Apple’s offering is awful and it needs a massive amount of work just to catch up to the better start-up players.
  • If Apple is serious about autonomous driving, I suspect it needs to acquire.
  • Zoox, Nuro and AImotive would be my top choices based on their performance (excluding Pony.ai which is Chinese).
  • Second: Viable independents: Zoox comes in at No. 3 with 1,922 miles driven for every disengagement up from 450 in 2017.
  • This is followed by Chinese player Pony.ai, US-based Nuro and Hungarian AImotive.
  • They still have a long way to go to produce a commercial solution but of the independents, these are the ones to watch.
  • Third: OEMs remain bad at autonomous driving: Top of the OEMs is Nissan which also ranked well in 2016 and 2017 but it is losing ground on the leaders with 52x more disengagements per mile driven than Waymo.
  • BMW, Honda, Toyota, SAIC and Mercedes-Benz are all way down on the list.
  • BMW is the best of the rest of the OEMs as it is 2423x worse than Waymo with Mercedes-Benz and SAIC being not far off 10,000x worse than Waymo.
  • Given the lengthening timeline for the commercialisation autonomous driving, I think that there will be plenty of opportunities for these players to purchase an off-the-shelf solution or acquire a struggling start-up with a good solution but no revenues.
  • GM’s acquisition of Cruise has put it in a very good position as a leader in autonomous driving.
  • Fourth, Uber: is the leader in ride-hailing but its offering has worsened materially since it last reported data.
  • In 2016 Uber reported roughly 1 disengagement per mile which worsened to 2.6 disengagements per mile in 2017 and 2.9 disengagements per mile in 2018.
  • Uber is now 31,570x worse than Waymo and I think it needs to acquire if it wants to have any future when the world goes autonomous.
  • I suspect that Uber’s worsening performance was due to the removal of any IP that was deemed to have been misappropriated from Waymo following Waymo’s suit.
  • It looks like I was wrong about Waymo’s IP helping Uber as the dire state of Uber’s home-grown offering is now plain to see.

Take Home Message.

• Waymo has seen a good improvement in performance but its disengagements still fall well below what would be acceptable as a commercial solution.
• Furthermore, the DMV data is not a reliable guide as the data could easily be skewed to look better than it really is.
• For example, Waymo or anyone could have just driven round and round the block on the same street thousands of times which would give much better figures than actually completing real journeys.
• Furthermore, this data is at odds with observations in Arizona where Waymo vehicles were observed struggling to do even the most basic tasks (see here).
• This would fit with the observations of deep learning scientists that have shown that even tiny changes in the data set (like moving from California to Arizona) can have a large and deleterious effect on the performance of an algorithm trained using deep learning.
• Hence, I think that the autonomous driving industry remains pretty far from having a working solution and I remain happy to stick with RFM’s 2028 target for commercially available autonomous vehicles.