Autonomous Autos 2021 – Creaking at the seams

AutoX takes a questionable lead.

• Waymo, Baidu (Apollo) and Zoox have fallen way down in the ratings leading me to question whether the analysis of the California DMV data as a measure of autonomous driving has finally passed the end of its useful life.
• California state law requires that all companies that test in the state report their performance to the Department of Motor Vehicles (DMV) on an annual basis and since 2017, this has provided a reasonably accurate picture of the state of autonomous driving.
• However, this analysis remains flawed as testing is now going on in many other US states and countries where no data is available.
• Furthermore, it is a relatively simple task to game the data as driving a vehicle round and round the same block 1,000 times will yield much better results than driving in downtown San Francisco or LA.
• Some testers also run their tests in the worst conditions possible as this is when their systems are most likely to fail and only by training in these conditions can improvements be made.
• Despite these issues, to date, the California DMV data has been in line with RFM’s research and anecdotal data that can be gleaned from the market, making it a pretty useful tool.
• This now appears to be breaking down meaning that I am losing confidence in this analysis’ ability to describe reality.
• 2021 California disengagement data indicates:
  • First, the leading group grows with there not being very much to differentiate the performance of the top 7 players.
  • These are AutoX, Cruise, Didi, Argo AI, WeRide, Deeproute AI and Pony AI and of these, 5 are of Chinese origin.
  • This is a further indication that progress towards autonomous driving is slowing down and that the effort that it takes to go from 95% to 98% is orders of magnitude greater than the effort required to go from 0% to 95%.
  • This is why I suspect that performance is starting to bunch together at the top end of the range as the laggards catch up.
  • Second, fallen leaders: Waymo, Zoox and Baidu (Apollo) are now way down the rankings being 6.3x, 66.3x and 78.7x worse than the current leader AutoX.
  •  I suspect that this has more to do with these companies testing elsewhere than relates to a sudden collapse in the quality of their technology.
  • For example, Waymo is running a semi-commercial robotaxi operation in Arizona for which there is no data.
  • Another possibility is that new features are tested and trained in California before they are deployed in other areas meaning that the technology is measured in its worst possible condition.
  • Nvidia is a good example of this.
  • Nvidia scores low down at 867x worse than AutoX but the company deliberately challenges its solution by training it in the toughest conditions possible.
  • Third, missing in action: Mobileye, Tesla and Yandex do not test in California and so there is no data that would allow them to be positioned on the leader board.
  • RFM research indicates that both Mobileye and Yandex have good quality offerings but cannot be ranked as they do not publish any data.
  • RFM has previously ranked Tesla in the bottom third, but this data is now too old to be of any real use and Tesla drivers say that the self-driving capability has improved materially over the last few years.
• The net result is that I think that the reliability of this analysis as a yardstick for the state of autonomous driving is now past its expiry date.
• Consequently, the only real conclusion that be drawn with any reliability is that the pace of improvement is slowing which is causing the leading offerings to all bunch together at the top.
• This underlines my long-held position that when autonomous driving hits a commercial grade, there will be a number of solutions on offer some of which may not have a natural home in a vehicle.
• Hence, OEMs may find themselves with several options to choose from meaning that it doesn’t make very much sense to try and develop a solution in-house.
• Instead, OEMs need to focus on ensuring that the user’s smartphone stays in his pocket when he gets into the vehicle as this is likely to be the only way that they will be able to earn decent revenues from digital services in the vehicle.
• This is the real challenge they face as I think that autonomous driving will take care of itself in time.