Baidu & AI – ChatCCP pt. II

Baidu highlights the CCP’s conundrum.

• Baidu’s latest model release claims to be better than ChatGPT but raises more questions than answers with opaque testing and an uncertain regulatory environment which may prevent this innovation from seeing the light of day.
• Internal testing by the state-owned China Journal of Science (see here) claims that Wenxin 3.5 (aka ERNIE) had surpassed ChatGPT and some other models on a series of standardised tests.
• ERNIE 3.5 was announced on the 26^th of May and is Baidu’s latest foundation model that is 260bn parameters in size representing a 26x jump from ERNIE 3.0 that was just 10bn.
• This is in line with the current thinking in large language models where performance has been demonstrated by Kaplan et al (see here) to be directly related to the size of the model which is why these models are getting bigger and bigger.
• ERNIE 3.5 is a foundation model which is where all LLMs begin their lives.
• From there they can be fine-tuned to perform other tasks with a much higher level of performance which is how OpenAI created ChatGPT from GPT3.5.
• This is why the analysis done by the China Journal of Science is not very well constructed because it is comparing both foundation models and models that have been further trained for other tasks against each other.
• A better comparison would be to leave ChatGPT and ChatGLM-6B out of the analysis and on that basis, it would be able (assuming the data is real) to claim that it is superior to GPT-4.
• This is a much more interesting claim to make because GPT-4 is supposed to be the latest and greatest that OpenAI has to offer but I get the feeling that the China Journal of Science is more interested in propaganda than science.
• This is why it headlines with an inferior claim both empirically and scientifically because it knows that this is the claim that everyone will understand and is far more likely to get attention and press coverage.
• This is unfortunate because I think it damages Baidu’s credibility which RFM has ranked as the global No. 2 in AI for the last 7 years and is a quality developer of AI in my opinion.
• This also raises a dilemma for the Chinese state which has a very difficult choice to make (which is similar to what is going on in Europe at the moment).
• China’s technology capability has been hammered by the limitations being placed upon its ability to manufacture advanced semiconductors and it is absolutely determined that this will not be repeated in other technologies.
• AI is one of the areas where it is competing on a level playing field and, in some areas (such as facial recognition and video categorisation), I would argue that it is ahead.
• This gives China a springboard from which to leap ahead in terms of generative AI but here the other side of the Chinese state gets in the way.
• ChatGPT, Bing, Bard and other generative AI systems can be easily induced into making bad or untrue statements and the Chinese state is concerned that this could be used to undermine its standing and authority with the Chinese people.
• This means that all generative AI systems have to be approved by the regulator before they can be released to the public.
• This approval requires the assurance that these systems will not behave badly but with the way these systems are designed, there is no real way this can be guaranteed.
• This is because all neural networks and especially LLMs are black boxes where their creators know that they work but have very little idea how.
• Regulators in China (and probably the EU) are not going to like this at all and they will either have to take a risk or enforce requirements that will hamper the performance of the LLMs.
• Given China’s apparent preference for the party over the economy, there is a very real chance that the Chinese state hampers the development of technology where it is also desperately trying to compete.
• This is also why there are concerns in the AI industry that draconian regulation in the EU may mean that AI leaves European shores to be developed elsewhere.
• The net result is that ERNIE 3.5 has made some impressive claims but has yet to be properly tested in the wild and it is only on that basis that it can be judged.
• I still think that Baidu has a shot at being a major player in AI as long as it can convince the regulator not to neuter its innovations before they leave the factory.
• Thanks to the truly awful sentiment surrounding Chinese technology, Baidu remains by far the cheapest way to invest in AI but because it is Chinese, no one wants to know.
• This is where I would look first when looking for a position in AI.