Waymo – Puff research.

A study that misses half the issue

• Waymo has released a study that extols the ability of autonomous driving to prevent fatalities on the road but as often with these sorts of studies, I don’t think that it is telling the full story.
• Waymo has published a study (see here) where it recreated the details of 72 fatal crashes that occurred within its Arizona operating area to see if the autonomous system could fare better than humans.
• The vast majority of these accidents involved two vehicles and, in these instances, Waymo put its self-driving system in the position of both the initiator and the responder.
• At a high level, the results were excellent where the autonomous system was able to avoid 100% of the accidents when it was both in the initiator and 82% of accidents in the responder role.
• In the responder role, the autonomous system was able to take evasive action that reduced the severity of the accident in a further 10% of the simulations.
• The last 8% of simulations were largely situations where the responder vehicle was stationary which is not a situation that a human or an autonomous system can reasonably be expected to avoid given the very limited opportunity to respond.
• This is being touted as evidence that machines are safer drivers than humans which one day will almost certainly be the case as 94% of accidents are caused by human error.
• However, I think that this is very far from being the case today and that reality is that machines are still far worse than humans at driving cars contrary to the conclusions that this study draws.
• Consider the following:
  • First, humans are very good at driving vehicles in general and this study completely ignores this side of the situation.
  • The study focuses only on what machines can do when humans get it wrong and not what humans can do when the machines get it wrong.
  • Disengagement data (see here) indicates that the best of the best autonomous driving solutions can drive for around 30,000 miles without a mistake that requires human intervention.
  • It is also worth noting that these miles are driven within a geographical limit that has been extensively mapped and consequently well known to the autonomous agent.
  • By contrast, humans drive on average 516,000 miles anywhere without making a mistake and 100,000,000 miles without killing anyone.
  • To be fair to the study, it is impossible to examine the other side of this situation, but I think it needs to be taken into account when putting the results of this study into the context f reality.
  • The net result is that this study, while anecdotally interesting, presents no further evidence of machines’ suitability to be let loose on the road.
  • Second, simulated data. The details of the accidents were recreated in a simulator and then used to test the autonomous agent.
  • The problem with a simulation is that it is by definition constructed from perfectly labelled data meaning that any edge cases will have already been captured and defined so that the machine doesn’t make any visual interpretation errors.
  • This is not an accurate representation of reality and it is well documented that the biggest problem with autonomous driving is not its decision making but its ability to accurately interpret the world around it.
  • This is because these visual systems are based on deep learning which is essentially a pattern matching system meaning that if it has not seen a particular pattern before, it can make the simplest errors that humans would never make.
  • Using a simulation removes this issue from an autonomous agent thereby making it appear to be far better than it really is in my opinion.
  • I have long been sceptical about the use of simulations to improve deep learning systems and have yet to see any evidence that it helps make these systems better.
• The net result is that this a really a marketing effort by Waymo which does remain one of the leaders in the field of autonomous driving.
• The gap between it and Cruise, Baidu and AutoX is now pretty small and so there is no reason to think that these systems would not also perform very well under the same conditions.
• Furthermore, there are plenty of other players such as Yandex or Mobileye for which there is no data but where they have demonstrated very good performance.
• Hence, while this enables Waymo to state that it could save lives, I think it does very little to accelerate the arrival of real autonomous driving.
• I continue to think that 2028 is the date when autonomy can go properly commercial.
• I still see no reason at all to any put money into this sector as there is already a massive oversupply of both capital and market participants.
• Some of the shine is already coming off this sector with both Tesla and the SPAC craze that it supports now comfortably in bear market territory.
• Rational humans should still steer well clear of this segment as valuations still have a long way to fall.