Artificial Intelligence – Scaling Debate

Scaling laws appear to be dying.

• There are more signs that the “scaling laws” that have underpinned the AI explosion (and all of the hype attached to super-intelligent machines) are coming to an end meaning that the real potential of LLMs is now visible and is falling way short of the craziest of forecasts.
• It is important to note that these new indications are anecdotal and as such do not represent any form of empirical proof, but they add to what is already being seen with existing models and how they are performing in reality.
  • First, an article from The Information: (see here) that claims that Open AI’s new models are not improving as quickly as expected and so the company is looking at new strategies to keep improving the performance of its new models.
  • Since its inception, Open AIs belief has been that with enough data and enough compute, artificial superintelligence would magically pop out at the end.
  • I have often referred to this as the “Infinite Monkey Theorem” (see here) and have held the opinion since 2020 (when I first wrote about LLMs (see here)) that this would not hold.
  • The radical underperformance of Open AI’s o1 model relative to what we were told is yet another sign that LLMs are beginning to experience the law of diminishing returns.
  • Second, commentary from an industry insider: who is the CEO of Deep Trading (algorithmic trading) who claims he was told that another one of the leading creators of LLMs has also hit a big wall of diminishing returns (see here).
  • This is even more tenuous than the article from The Information, but it adds weight and a second unrelated “data point” implying that LLMs are beginning to reach the limits of what they are capable of.
• Diminishing returns is a huge problem because it means that one has to use increasing amounts of resources in order to achieve smaller and smaller improvements.
• This very quickly becomes uneconomical, and any system that is funded by private money soon finds that willingness to pour more money in quickly dries up.
• This could easily trigger a correction of expectations which in turn would cause valuations of the most outlandish companies to fall meaningfully.
• It is my opinion that diminishing returns have been evident for quite some time supported by the fact that there is not much difference between the big models today in stark contrast to 2 or 3 years ago.
• It is the slowing improvements that allows the laggards to catch up which is why when one looks at the benchmarks these days, the differences are minor.
• LLMs still have substantial use cases that will deliver great economic benefits and spawn a new industry, but superintelligence is as far away today as it was 10 years ago.
• The LLM superpowers of using natural language as a man-machine interface and the ability to ingest categorise, cross reference and regurgitate unstructured data remain very much intact and when properly used will be extremely valuable.
• These two abilities alone open up many possibilities for new businesses as well as the replacement or improvement of businesses that already exist.
• I am not bearish on the outlook for LLMs but merely cautious on valuation as expectations have run far ahead of what is realistically possible meaning that a correction is needed to bring expectations back to reality.
• The robots are not coming to kill us anytime soon, but they will be making an appearance in the economy in ways that will make digital life for users better and more productive as well as allow companies to make far better use of the data that they already have but have forgotten about.
• There is a correction coming and the time to invest will be when everyone has given up on AI and moved on to the next bright and shiny theme.