Google Deep Mind – Inching forward

Deep Mind chips away at the big challenges.

• Deep Mind has released new research on language models which is an attempt to address 2 of the big problems with deep learning and again demonstrates that it is one of the best AI companies in the world.
• Deep Mind has published 3 papers (see here, here, and here) that outline the creation of a new model that is much smaller than OpenAI’s behemoth (GPT-3) but is able to perform better and offer insight into how the algorithm is arriving at its answers.
• Two of the big problems (there are plenty more) are:
  • First, data quantity: Deep learning systems require vast quantities of data in order to be able to correctly identify statistical patterns in the dataset they are fed.
  • This makes the training of deep learning networks very expensive in terms of data collation as well as the compute power that is required to process all this data.
  • OpenAI’s GPT-3 uses 175bn parameters while Microsoft’s Megatron uses an incomprehensible 530bn parameters.
  • This problem limits both the accessibility of deep learning and its applicability outside of massive data centres.
  • Hence, anything that can reduce the amount of data needed is likely to have a profound effect on the affordability and the usability of deep learning.
  • Second, verification: The neural networks that make up deep learning systems are black boxes where one can see the inputs and the answer, one has no idea how the machine arrived at the answer.
  • For systems where safety is crucial (autonomous driving), not knowing how the system is working is a red line because there is no way to determine how reliable the system actually is.
  • Hence, any system that introduces better visibility into how it is working will allow deep learning to be more widely used and also better understood so it can be improved upon.
• Deep Mind’s answer to GPT-3 and other language models is called RETRO which is much more structured than the typical large black box of other deep learning systems.
• RETRO is split into a retrieval database where Internet knowledge is stored and updated and a transformer containing 7.5bn parameters that is pretrained on the database.
• Deep Mind was able to demonstrate superior performance to GPT-3 and its own in-house version called Gopher which had an also massive 280bn parameters.
• This is another demonstration supporting my opinion that the challenges of AI will not be solved by brute force.
• The OpenAI camp (including Elon Musk) subscribe to the idea that with enough data enough computing power, all of the limitations of AI’s performance can be solved.
• Microsoft and Amazon obviously love this because they sell the compute power that is used to crunch and store all of this data.
• However, what is being increasingly demonstrated is that a more sophisticated approach where the system can work out where to search rather than searching everything performs better with less data and consumes fewer resources.
• Deep Mind is not the first to demonstrate this as there are other approaches that combine deep learning and rules-based software which have also had promising results.
• I continue to think that this is how the autonomous driving problem is going to be solved as well as improving the performance and affordability of other systems.
• Deep Mind may lose a lot of money, but this increasingly looks like $500m well spent.