Artificial intelligence

Chinese girl idol group SNH48 creates digital clones built by AI

  • China’s pop music fan economy is estimated to be worth $14.5 billion by 2020
PUBLISHED : Tuesday, 25 December, 2018, 6:01am
UPDATED : Tuesday, 25 December, 2018, 6:33am

A new generation of idols are singing and dancing in music videos in China, with plans to sell albums and perform in concerts where they will engage fans with personalised interaction. Only thing is, they don’t actually exist, at least corporeally.

In the latest Christmas music video released by Chinese girl idol group SNH48, six of the group’s most popular stars sing and dance with some special partners – digital copies of themselves based on their looks, voices and body language.

The four-minute music video, co-produced by Tencent-backed artificial intelligence (AI) start-up ObEN, claims to be the world’s first commercially released song co-starring human singers and their AI 3D avatars.

“This song is our first step to test the waters in the virtual idol market. We are planning to create more intelligent virtual idols, releasing albums and making movies for them,” said Xiong Wei, vice-president of the Shanghai-based SNH48.

“Human idols are hardly perfect. They have their limits in terms of age and there may be gossip about their love affairs. None of these will be an issue for virtual idols,” said Xiong, adding that a female idol usually has a career lifespan of eight years before she stops “exuding a youthful glow”.

Similar to Japanese idol girl group AKB48, SNH48 features more than 100 female performers aged around 20 who perform regularly at the group's own theatre and interact with fans predominantly via live promotional events.

By using computer vision, natural language processing and speech technology, the Pasadena, California-based ObEN is able to create 3D intelligent avatars that not only look, sound, and learn to behave like their individual human counterparts but can stay forever young.

In China’s entertainment business the days of pirated albums are history, with millennial fans becoming increasingly generous in showing support for their idols. China’s pop music fan economy is estimated to reach 100 billion yuan ($14.5 billion) by 2020, according to Zhongtai Securities.

SNH48’s partnership with ObEN is not the only one exploring the potential of AI in the entertainment industry. AI Stars, a joint venture with S.M. Entertainment, South Korea’s largest entertainment agency, claims to be the first celebrity AI agency, combining the intellectual property of SM’s top k-pop stars with ObEN’s AI technology to create new interactive experiences and products.

In the music video, which features the song Now and Forever, the solo performance is handled by the real SNH48 stars while the chorus is sung by their AI clones. The dance moves show the interaction between the human singers and their digital copies.

To create digital clones of the singers, ObEN used photographs and voice recordings, with each singer required to read 100 to 200 sentences which were used as sample data for the AI algorithm. The result was a digital avatar that could sing like each group member and even speak multiple languages in their own voices, according to Adam Zheng, chief operating officer and co-founder of ObEN.

“Our personal AI technology has great potential in the entertainment industry. Let’s say you have an actor who stars in a movie. A producer may want him to sing the ending theme song, which he is not good at. We can make his AI avatar sing for him,” said Zheng.

To make a more precise digital copy, a 3D body scan would be needed but that was not done in the case of the SHN48 avatars.

There are already virtual stars in the music industry. Hatsune Miku, the schoolgirl-like, turquoise-­haired anime mascot is arguably the most famous. Designed by a Japanese software company, she was first used to sell a voice synthesizer. As consumers began using the product to compose original music, Miku became a sensation, drawing massive numbers of followers and even performing concerts where her hologram “sings” fan-written songs.

Can China’s new AI news anchors give Anderson Cooper a run for his money?

Last month China’s state-owned Xinhua News Agency introduced so-called “composite anchors”, using AI technology provided by Beijing-based search engine operator Sogou to combine the images and voices of human anchors.

Digital avatars like Miku and the Xinhua newsreader are not “intelligent” in that they can communicate with people, but Xiong said in future the technology might be able to make the NH48’s virtual idols capable of personalised interaction with fans.

For now, that capability is limited to more simple functions like sending out personalised wake-up calls to fans, though another advantage is the ability to have digital group members appear in different places at the same time.

“Our stars are busy travelling across China to perform 1,000 shows a year in different cities so the virtual idols can sing and dance in different locations at the same time,” said Xiong.