Artificial intelligence
Get more with myNEWS
A personalised news feed of stories that matter to you
Learn more
The demo version of the XiaoK Show mini program allows you to upload a dancing video and pick a cartoon character to bust out all the same moves. (Picture: YunBo)

The company turning TikTok moves into avatars, without a depth sensor

YunBo’s tech can already be found in games from Tencent and Perfect World, but it’s bringing motion capture to the masses with a new mini program on WeChat and Douyin

This article originally appeared on ABACUS

Imagine if you could customise your Fortnite avatar to bust out the exact same dance moves you put on TikTok with just a click of a button. For some Chinese games, this is already happening thanks to an AI company called YunBo.

Motion capture, once reserved for professionals with expensive gear, has become much more accessible in recent years. But even something like Animoji on the iPhone relies on depth sensors to get a 3D map of subjects.

Now YunBo says it has an algorithm that doesn’t need a depth sensor, and it will soon bring its new technology to the masses with a new mini program.

The mini program, called XiaoK Show, will launch on WeChat and Douyin, China’s version of TikTok, by the end of March. But I recently got to give the demo a try. The app lets users record themselves dancing and turns it into a virtual character showing off all the same moves within minutes.

Tencent’s Pokémon Go clone Let’s Hunt Monsters already incorporates YunBo’s technology. (Picture: Tencent/Bilibili)

XiaoK Show appears to be a simple utility app, but it does what it promises. To get started, you upload a short dancing clip and select the cartoon character you want to mimic your moves. You can then download the video and share it on other social media platforms.

The demo wasn’t especially polished and the number of characters and backgrounds available was limited. But the results are impressively accurate.

The demo version of the XiaoK Show mini program allows you to upload a dancing video and pick a cartoon character to bust out all the same moves. (Picture: YunBo)

The most impressive part of XiaoK Show, though, is that it can run on any phone with WeChat or Douyin. And that’s just about every smartphone in China these days. Without the need for a depth sensor, people don’t need to buy a US$1,000 iPhone to try out this motion capture tech.

“Our algorithm is smart enough that you don’t even need a depth sensor anymore,” YunBo co-founder Mei Song said. “It can just simulate depth on its own.”

The algorithm is also able to generate depth data that can be stored in a file. Mei said extrapolating depth data from any recording is what sets YunBo’s algorithm apart. This means anyone can grab a video online and apply it to their own avatar, he added.

This tech has also already shown up in a few popular games in China. Tencent’s Pokémon Go wannabe Let’s Hunt Monsters and Perfect World’s eponymous MMORPG have included YunBo’s algorithm, allowing users to upload their own dance videos to animate their in-game avatars.
Mei Song said the success of XiaoK Show depends on whether he can license the likeliness of famous characters to be used in the app. (Picture: YunBo)

Before starting YunBo three years ago, Mei and his co-founder Chen Min worked at Chinese gaming heavyweight Linekong Interactive as senior VP and CTO respectively. But after 10 years at the company, the two left to start their own business so they could focus on creating technology instead of designing games.

And that’s what they’re doing now with XiaoK Show, which Mei said he hopes will “turn everyone into an animator”. A full-fledged app is also in the works, Mei said.

Another product the YunBo is working on might also help bring their motion capture tech to a new audience. This product will allow people to live stream as a virtual avatar, which has recently become a trend in China. A Shanghai college professor, for example, found he could engage his students more in online classes while teaching in the form of an anime girl.
If you want a taste of how YunBo’s algorithm works for yourself, the company offers another small demo on its website that lets users turn a 30-second dance video into an animation project. You can view the results either on the website as a Unity project or download it as an FBX file. This can then be imported into a professional 3D graphics program like Maya.
YunBo’s demo online can be exported as an FBX file and opened in professional animation programs. (Picture: YunBo)

Mei acknowledges that the motion capture business is bound to be competitive going forward. But he believes that his company, which currently has just a dozen employees, is in a good position to make motion capture popular among average smartphone users.

“I wouldn’t say that nobody can do what we do,” Mei confessed. “But we definitely got in early and are ahead of the pack.”

Purchase the China AI Report 2020 brought to you by SCMP Research and enjoy a 20% discount (original price US$400). This 60-page all new intelligence report gives you first-hand insights and analysis into the latest industry developments and intelligence about China AI. Get exclusive access to our webinars for continuous learning, and interact with China AI executives in live Q&A. Offer valid until 31 March 2020.