Check out this sign language-translation glove that gives mutes a voice everyone can 'hear'

Imagine a device which translates hand gestures into spoken word. we meet three Hong Kong students who are changing the way users of sign language communicate

Young Wang |

Latest Articles

STUDY BUDDY ANSWERS: Unfazed by dirty work, Hong Kong university infectious disease doctor leads frontline fight against Covid-19 [September 30, 2020]

Hong Kong students discuss going back to school during Covid-19

Malvern College prepares students to succeed as future leaders

(From left) Jacky Chan, Noddy Chung, and Michael Ng with their “special” gloves.

Being heard is often a struggle for users of sign language, but a new invention by three students at Christian & Missionary Alliance Sun Kei Secondary School hopes to change that. Sign-translate gloves can translate sign language into text and speech through a smartphone app, bridging the gap between sign and verbal communication.

The team behind the invention are Form Five students Noddy Chung Ho-fai, Jacky Chan Ka-lung and Michael Ng Chak-lam.

After coming top in the Senior Division of the Hong Kong Student Science Project Competition in April, they went on to win the Bronze Award in the Engineering category of I-SWEEEP 2017, an international competition held in the United States which challenges students to tackle global issues.

On their return to Hong Kong, the young innovators explained to Young Post how their invention works.

Wearing one of the prototype gloves, Jacky points at five flex sensors on each of his fingers: as the sensor is flexed, the resistance across the sensor increases and data is collected based on how much his finger bends. At the back of the hand, a gyroscope detects acceleration and angle of the hand. The data collected is then transmitted through a bluetooth chip also attached to back of the glove. Upon receiving the data, a breadboard - used for making an electric circuit - translates the digits into ordinary language, which can be presented on the app as text and even speech.

Clearly, the knowledge applied to this invention exceeds the level of the average secondary school Physics class. Instead, the boys learned from teachers and sifus around them as well as the Internet.

Among the three of them, only Noddy takes Physics as a HKDSE elective; Michael studies humanities subjects while Jacky opted for business and economics. Nevertheless, they’ve found a way to combine each of their strengths in the project. Jacky prepares for meetings with potential investors while Michael oversees the paperwork, designing banners, preparing presentations for competition judges, and making booklets in both Chinese and English detailing every aspect of the project.

“Our team hopes to change the situation of the isolation of deaf-dumb people through these gloves,” reads the booklet.

Having worked out the principle of the gloves, the next challenge for the team was finding the materials. The flex sensors, one of the most important parts , had to be sourced online. Cost was also a factor: just one sensor costs over HK$100. With ten sensors needed for one pair of gloves, the cost adds up to HK$1,500.

(From left) Jacky Chan, Noddy Chung, and Michael Ng with their “special” gloves.
Photo: Dickson Lee/SCMP

The overall budget for the project exceeded HK$5,000. After a HK$2,500 school grant, the boys dug into their own pockets for the rest.

“We just paid for whatever parts we ordered,” says Jacky.

They worked on the project non-stop for three months, staying up every evening after school until 10pm. It hasn’t been smooth sailing.

“We’ve failed at least ten circuit boards,” remembers Noddy. Their app has also undergone more than 50 updates. The team would even pull all-nighters ahead of competition deadlines, and “go home at 6.30am to take a shower before school,” says Jacky.

It all paid off, though, when they completed the first prototype. “It was liking watching the birth of your own child,” Jacky recalls.

Unable to sign themselves, the boys rely on Chinese University of Hong Kong’s Sign Language Browser, which has a directory of thousands of words in one of Hong Kong’s most-used sign languages.

For now, Michael explains, the app’s output to text and speech is only in English, so that it can be used when travelling overseas and generally have a wider outreach. But the team eventually plans to expand their database to include French. Meanwhile, sign language users can also input words themselves, to suit their own vocabulary usage.

Initially, the team manually input all the data, until they decided to write an app that does the work for them. However, it’s still a long process, Noddy explains, because the app needs to be able to recognise slight variations in hand gestures as belonging to the same sign. With more than 5,000 data inputs, there are still only around 20 words in their database at the moment.

“We want to input more words so we can get some sign language users to test it out,” says Michael, with Jacky adding that they hope to improve their invention with user experience feedback.

Jacky also hopes to receive funding from the government or independent groups to lower the cost.

“We’re not looking to commercialise the product. We want to help the people in need.”

Edited by Charlotte Ames-Ettridge