Advertisement
Advertisement
E-commerce
Get more with myNEWS
A personalised news feed of stories that matter to you
Learn more
Deaf live-streamer Shaoman co-hosts a video shopping session with fellow host Chengxiao CC on Alibaba's Kaola e-commerce platform. Picture: Screeshots of Kaola

Alibaba experiments with basic captions for deaf live-streaming e-commerce sellers

  • Kaola, a cross-border e-commerce platform, is one of the first in China to feature live shopping videos hosted by deaf streamers
  • Real-time machine translation of sign language remains a difficult challenge
E-commerce
Live-streaming has become such a normal part of the online shopping experience in China that even the government started recognising “live-stream salesperson” as an official profession this year. Now one of Alibaba Group Holding’s e-commerce platforms says it wants to help people who are deaf or mute get into the business.

Alibaba is the parent company of the South China Morning Post.

Live-streaming shopping sessions are typically full of chatty sales pitches. But in a nearly two-hour session on Thursday, two women with hearing impairments co-hosted a relatively quiet live stream on Kaola, an online platform for overseas brands selling to Chinese shoppers.

The guest hosts were also joined by a Kaola live-streamer who spoke, and together they showed off products ranging from mixed nuts to ready-to-eat French fries. For viewers, captions appeared above the hosts’ heads summarising what was being expressed in sign language.

Some local media dubbed it a “sign language recognition system”, but it did not appear to have been performing sophisticated real-time translation. Rather, the captions showed words and short phrases such as “tastes good” or “sweet” around thirty seconds after they were signed.

The words for the captions first appeared as pinyin, the common romanisation system for Mandarin Chinese often used for typing, before they were rendered into Chinese characters. Alibaba did not immediately respond to questions about the technology.

Deaf courier team delivers good news for disabled workers in China

The streaming session took place on December 3, the United Nations’ International Day of Persons with Disabilities, but it is not the first of its kind on Kaola. One of the co-hosts, Shaoman, was the first deaf live-streamer on Kaola earlier this year.

Originally working as a “daigou” – an agent who flies abroad to shop on behalf of customers – Shaoman’s business collapsed when the coronavirus pandemic halted overseas travel, Chinese news outlet The Paper reported. Her husband, who painted handheld fans for a living, also lost his job.

Looking for a new gig, Shaoman signed with Kaola to become a live-streamer. But since not all words have an existing equivalent in sign language, she had to get creative.

Names like Kaola, for example, had to be invented. With the help of a sign language teacher, Shaoman settled on the sign for koalas – Kaola’s mascot. She made her live-streaming debut on September 27, the last day of the International Week of the Deaf this year.

Even though most of the session was conducted in silence, the stream reportedly drew thousands of viewers.

03:24

China’s delivery business provides a career path for the deaf

China’s delivery business provides a career path for the deaf

Machine translation of sign language has been an ongoing challenge worldwide because it requires a combination of continuous hand gestures, movements and sometimes facial expressions. Hands often block each other in sign language, making it difficult for computers to see what is going on. An additional challenge is that, unlike spoken or written languages that separate words and sentences with pauses and punctuation, there is often no clear distinction between where one sign ends and another begins.

Last year, researchers from Google’s AI lab said they developed a new system to track the position of about 20 coordinates assigned to fingertips, knuckles and parts of the palm. The relatively small amount of data collected allows the algorithms to operate in real time while using the limited computing power that might be available on a mobile phone.

Google researchers also said they were hoping to eventually support recognition of hand movements on top of static gestures.

Post