China education
Get more with myNEWS
A personalised news feed of stories that matter to you
Learn more
Media filming AI-MATHS as it sat the test in Chengdu. Photo: Xinhua

Robot scores in China’s national maths exam ... but it’s still not good enough for a top university

Machine exceeded expectations, getting 70 per cent of the questions right, but continues to struggle with the subtleties and ambiguities of language

A robot has taken the maths sections of China’s notoriously difficult national college examination and got 70 per cent of the answers right, state media reported.

The AI-MATHS robot, which is about the size of a refrigerator, took the exam paper in a technology park in Chengdu in Sichuan province on Wednesday.

Under the supervision of government inspectors, it spent more than 20 minutes to complete the questions ranging from algebra to geometry and achieved a score of 105 out of the total 150, the reports said.

AI-MATHS was built by Zhunxingyunxue, a Chengdu-based cloud computing company jointly owned by Tsinghua University in Beijing.

One of the first – and easiest – questions was, if the complex number (1-i) (a+i) corresponds to the second quadrant in the complex plane, what is the value of the real number (a). The exam robot provided a correct answer, according to teachers who marked the paper.

AI-MATHS did get stuck on simple sentences, though. It could not understand, for example, what “the number of teachers was twice more than male students” meant.

Professor Sun Maosong, director of the Natural Language Processing Group at the department of computer science and technology at University, said the performance of the artificial intelligence in the exam exceeded his expectations.

The technology could find a use in a wide range of applications, including education and medical diagnosis, he said.

However the robot struggled to understand some language with its subtleties and ambiguities, he said.

The exam robot fell far below the level required by top-tier universities in China such as , which usually ask for a score from prospective students of 140 or higher.
Staff monitoring the test in Chengdu. Photo: Xinhua

The machine, however, beat the score of about half the high school students in terms of the understanding and solving mathematical questions, said Sun, who was not directly involved in the project.

A maths test forms part of the annual national college entrance exam in China, known as the gaokao, one of the toughest of its kind in the world.

The robot’s score provides a good benchmark to evaluate the recent advances in artificial intelligence against the human mind, according to Sun.

The challenge in the exam was not simply mathematics, but also understanding the language used to phrase the questions, he said.

“When a robot can achieve 130 or beyond, it may be regarded as truly intelligent, but I’m afraid it will take many, many years of effort to reach that level,” he added.

The robot scored 93 points out of 150 in a similar test in February, lower than the student average of 106, according to the state-run news agency Xinhua.

Lin Hui, the chief executive officer of Zhunxingyunxue, told the news website the machine could have performed better.

“If we could have an hour and a half, the score could be increased by 10 points or more,” he was quoted as saying.

Qi Zuhai, one of the three high school teachers who graded the robot’s test, told the China News Services he was surprised by the computer’s handling of some questions.

“There was a three-dimensional geometry question. The robot gave a brilliant solution totally out of the box,” he said.

The gaokao robot is part of a national science project on artificial intelligence funded by the central government.

More than 30 research teams from universities and research institutes, including the Chinese Academy of Sciences and the University of Science and Technology of China, have taken part in the project since 2015. The purpose was to develop an AI system capable of understanding and solving most questions posed using language.

Sun said mathematics was just the beginning.
A printout of the steps taken by the computer to work out a geometry question. Photo: Xinhua

Different versions of the robot such as AI-Chinese, AI-Geography and AI-Physics will take sections of the gaokao test, possibly starting next year.

“It is possible for a machine to write an essay,” he said. “I expect it to achieve a score of 80 [out of 100],” Sun added.

Scientists in Japan have also developed an exam robot called Todai which has taken the Japanese college entrance exams many times.

By the end of last year, however, the researchers dropped the project after the robot failed to achieve its goal of achieving a grade to get into the University of Tokyo.

This article appeared in the South China Morning Post print edition as: Robot fails to make grade on maths test