Baidu’s Ernie Bot tops Chinese large language model rankings by Xinhua think tank, but lags OpenAI’s ChatGPT

Ernie Bot performs better in a series of tasks than competing services from Alibaba, iFlyTek and SenseTime, according to Xinhua Institute
A different LLM test in China finds Qihoo 360’s Smart Brain to be the best-performing Chinese model, followed by iFlytek’s SparkDesk

Artificial intelligence

Ben Jiangin Beijing

Published: 7:00pm, 12 Jun 2023

Why you can trust SCMP

Baidu’s Ernie Bot ranked top among ChatGPT alternatives offered by Chinese tech companies, according to a recent test conducted by Xinhua Institute, a think tank affiliated with the Xinhua state news agency.

The internet search giant’s chatbot, launched in March, led Alibaba Group Holding’s Tongyi Qianwen, voice recognition firm iFlytek’s SparkDesk, and image recognition company SenseTime’s SenseChat in Xinhua’s test of large language models (LLMs). Alibaba owns the South China Morning Post.

However, Baidu’s LLM still trailed US rival OpenAI’s GPT-4 and GPT-3.5, which are officially unavailable in mainland China. GPT-3.5 powers the original version of ChatGPT that was released to the public in November last year.

08:54

Is China’s technology falling behind in the race for its own ChatGPT?

Xinhua’s test measured general capabilities, including basic language skills – such as the ability to understand prompts and generate responses – and intelligence, which involves common sense, logical reasoning and subject knowledge in fields like mathematics, physics, finance and literature.

The test also gauges the chatbots’ ability to enhance the productivity of professionals such as journalists, painters, designers, marketing specialists, lawyers and researchers.

Chinese technology firms have been rushing to introduce competing chatbots after the launch of ChatGPT sparked renewed interest in the potential of artificial intelligence (AI), in particular LLMs – deep-learning algorithms that can recognise and generate text, images and videos.

A different LLM test by Clue, a Chinese website that follows AI research development, found cybersecurity firm Qihoo 360’s Smart Brain to be the top-performing Chinese model, followed by iFlytek’s SparkDesk. Both ranked below GPT-4, GPT-3.5 and US start-up Anthropic’s Claude.

Xinhua Institute noted in its report that its test was subject to time and conditional constraints, which may result in “a certain degree of subjectivity”.

Visitors at the iFlyTek booth during the 2019 Smart China Expo in Chongqing, China. Photo: VCG via Getty Images

Some Chinese experts believe that overseas companies are likely to still reign on the global AI stage when it comes to cutting-edge technology. Building up local competitors to ChatGPT will face challenges, such as gaps in training models, the availability of data sets, as well as the uniqueness of the Chinese language itself.

Baidu founder and CEO Robin Li Yanhong suggested in an interview with Chinese tech news outlet 36Kr in late March that Ernie Bot was “two months behind” ChatGPT. He later clarified that it could take the company any amount of time – from a short period to “forever” – to fill that two-month gap.

Li said earlier that month that Ernie Bot, which was the first ChatGPT-like service offered by a major Chinese tech firm, could “evolve faster with human feedback”.

At SparkDesk’s launch event in early May, iFlyTek founder and chairman Liu Qingfeng said the company’s chatbot would catch up with ChatGPT by the end of October.