The company turning TikTok moves into avatars, without a depth sensor

Imagine if you could customise your Fortnite avatar to bust out the exact same dance moves you put on TikTok with just a click of a button. For some Chinese games, this is already happening thanks to an AI company called YunBo.

Motion capture, once reserved for professionals with expensive gear, has become much more accessible in recent years. But even something like Animoji on the iPhone relies on depth sensors to get a 3D map of subjects.

Now YunBo says it has an algorithm that doesn’t need a depth sensor, and it will soon bring its new technology to the masses with a new mini program.

The mini program, called XiaoK Show, will launch on WeChat and Douyin, China’s version of TikTok, by the end of March. But I recently got to give the demo a try. The app lets users record themselves dancing and turns it into a virtual character showing off all the same moves within minutes.

XiaoK Show appears to be a simple utility app, but it does what it promises. To get started, you upload a short dancing clip and select the cartoon character you want to mimic your moves. You can then download the video and share it on other social media platforms.

The demo wasn’t especially polished and the number of characters and backgrounds available was limited. But the results are impressively accurate.

The most impressive part of XiaoK Show, though, is that it can run on any phone with WeChat or Douyin. And that’s just about every smartphone in China these days. Without the need for a depth sensor, people don’t need to buy a US$1,000 iPhone to try out this motion capture tech.

“Our algorithm is smart enough that you don’t even need a depth sensor anymore,” YunBo co-founder Mei Song said. “It can just simulate depth on its own.”

The algorithm is also able to generate depth data that can be stored in a file. Mei said extrapolating depth data from any recording is what sets YunBo’s algorithm apart. This means anyone can grab a video online and apply it to their own avatar, he added.

This tech has also already shown up in a few popular games in China. Tencent’s Pokémon Go wannabe Let’s Hunt Monsters and Perfect World’s eponymous MMORPG have included YunBo’s algorithm, allowing users to upload their own dance videos to animate their in-game avatars.

Before starting YunBo three years ago, Mei and his co-founder Chen Min worked at Chinese gaming heavyweight Linekong Interactive as senior VP and CTO respectively. But after 10 years at the company, the two left to start their own business so they could focus on creating technology instead of designing games.

And that’s what they’re doing now with XiaoK Show, which Mei said he hopes will “turn everyone into an animator”. A full-fledged app is also in the works, Mei said.

Another product the YunBo is working on might also help bring their motion capture tech to a new audience. This product will allow people to live stream as a virtual avatar, which has recently become a trend in China. A Shanghai college professor, for example, found he could engage his students more in online classes while teaching in the form of an anime girl.

If you want a taste of how YunBo’s algorithm works for yourself, the company offers another small demo on its website that lets users turn a 30-second dance video into an animation project. You can view the results either on the website as a Unity project or download it as an FBX file. This can then be imported into a professional 3D graphics program like Maya.

Mei acknowledges that the motion capture business is bound to be competitive going forward. But he believes that his company, which currently has just a dozen employees, is in a good position to make motion capture popular among average smartphone users.

“I wouldn’t say that nobody can do what we do,” Mei confessed. “But we definitely got in early and are ahead of the pack.”