In warm-up for an AI Celine Dion, AI choir and dancers accompany human orchestra in Hong Kong concert

As the lights dimmed and the audience settled into their seats in the Hong Kong City Hall Concert Hall in Central on July 14, superficially it seemed as if Hong Kong Baptist University’s (HKBU) symphony orchestra would begin its annual gala concert like any other.

However, the players were joined by some surprising companions – namely, a 320-voice AI choir and several AI ballet dancers who twirled on a big screen as the human musicians performed.

This wasn’t just a gimmick. The virtual singers and dancers were the public’s first glimpse of a university AI (artificial intelligence) project that has a serious – some might say alarming – mission: to translate what it means to be human into computer code so that machines can learn how to behave like emotional beings.

The (invisible) choir created by the University’s Augmented Creativity Lab joined the human orchestra in a rendition of Pearl of the Orient, Lo Ta-yu’s popular 1986 song about Hong Kong.

The computer-generated voices sang live in response to the baton of music director Johnny Poon Ming-lun, and were accompanied by a video backdrop that was also created by AI based on the lyrics.

Professor Guo Yike, university vice-president in charge of research and development, told the Post the fundamental goal of the Augmented Creativity Lab is not to build art tech that directly copies the human experience, but rather to develop technology that can harness its own creativity and, ultimately, “understand humanity”.

5 ancient Chinese instruments come alive in rousing Hong Kong concert

“We believe the machine can have its own creativity rather than mimic humans. That’s our major philosophical belief,” he says.

In 2021, as the Hong Kong government promoted the idea of turning the city into a major centre for “art-tech” research, the university received a HK$52.8 million (US$6.7 million), five-year research grant to develop AI technology that can transform the creative and cultural industries. The AI choir is an early product of the research.

Voice samples of the late Canto-pop singer Leslie Cheung Kwok-wing and eight other singers were used to train computer programs to emulate human singing based on sound waves, frequency and pitch. Then human voices were used as references to build up a 320-voice virtual choir.

“AIs are almost like kids, we have to teach them to expand their range. If you don’t feed them with enough data for a certain range, then the voice [will] start to crack, to sing out of tune, just like anyone else,” Poon says.

After the AI singers “learned” how to sing the song, the second part of the challenge involved the live performance.

Poon, who is the research project’s deputy coordinator, wore a motion capture suit on stage so the AI “singers” could read and interpret his movements and perform accordingly, whether to sing louder or lower, pause or stop.

A similar development process was applied to create the AI ballet dancers.

Aside from using body sensors to code human dance moves, the AI also took inspiration from videos of a newly discovered species of box jellyfish – first recorded by HKBU biology professor Qiu Jianwen and his research team – to develop its own choreography, so the resulting movements could emulate those of the jellyfish and match the waves of the ocean.

Guo admits that AI research brings up plenty of ethical and legal questions that have yet to be addressed.

“Any kind of technology will imply certain misuse,” he says. “You will always have technology [advancement], and then you always will have the risk.” That’s why, he says, it’s important for there to be a legal system that can apply discipline and mitigate the risks.

“We need to be very, very careful, and we have a responsibility,” Poon adds, noting that ethical considerations must be thorough and rigorous.

“Every time you use human subjects, every time when you use things that might infringe on one’s privacy, we need to hold that to the strictest standard,” he says.

For the voice samples and other references they used to train the AI choir and dancers and produce the video backdrops, explicit permission was granted by the original artists or rights owners, unless it was material available for free use in the public domain, he explains.

So will AI artists take over from human artists? Guo says: “I don’t believe machines will replace humans, unless humans don’t really improve themselves.”

He suggests that people should view AI as an additional tool that humans can learn to use as an extension of what is possible as opposed to a limitation.

After all, there will always be an evolution of machinery, Poon says.

He points to the development of the camera in the mid-19th century as an example: “When that first started, some painters already said, ‘Oh, that’s the death of painting’! But that’s not it. Painting is still treasured by so many people … The thing is, [technology] creates new opportunities – creates a new way to create art.”

From a social and cultural perspective, technology also allows people to capture moments of history and to partake in cultural preservation, Poon says.

“Think about the inheritance, think about the lineage, think about passing down history. Preserving something that is so close and dear to our heart – technology serves that purpose.”

There are many other AI initiatives from the researchers on the horizon (Guo hopes they can present an AI voice that sounds like Canadian singer Celine Dion by Christmas). What ties them together is the emphasis on using technology to develop and advance creativity.

“We always still hold human creativity in the highest regard,” Poon says. “This kind of technology, it’s not a competition, but I sometimes describe it as holding up the mirror and looking at ourselves.

“I remember the first time when we were still training the AI choir, the first time when we [had] something close to a real choir sound, I was literally moved, not because of what we have accomplished, [but] because I’m moved by human voices, machine learning from us.”