How Google Glass can improve autistic children’s social skills by reading facial expressions

Google Glass – the once globally hyped smart glasses – seemed to have slipped off the radar after sales were suspended in 2015, just three years after they were launched. Now it is being hailed as a life-changing device for people with autism spectrum disorder (ASD).

People with autism have trouble with social skills, and verbal and non-verbal communication.

Researchers at Stanford University have harnessed Google Glass to develop a form of self-guided therapy that families can use to coach an autistic child to read emotions in faces, ultimately improving their ability to interact with others.

Catalin Voss is the founder of the Autism Glass Project. The Stanford School of Medicine graduate student spoke at the EmTech Hong Kong 2018 conference in Hong Kong this month about his team’s augmented reality therapy that taps the Google Glass technology.

“The goal is to give a learning aid to kids and families,” explains the entrepreneur, who had previously sold his start-up Sension, a face- and eye-tracking-based innovation that discerns facial expressions, to a Toyota-owned company.

Voss and his team tweaked the concept for use on donated Google Glass sets and tested it on around 150 kids with ASD in a handful of small-scale studies.

“In general, it has improved eye contact, increased emotional recognition abilities and increased engagement in the family,” he says.

First Hong Kong run for play specially adapted for autistic children

One of those trials involved 14 families who used the technology at home over a six-week period. The participants used the device for 20 minutes, three times a week. Researchers then analysed participants’ interactions.

“What we found is more complex than just an emotional intervention,” says Voss. Initially the researchers thought the technology would teach kids how to better recognise emotions such as “happy” and “sad”, but they discovered the process also sparked more conversations within the family about emotions.

“It helped kids realise there is something for them to see in faces, so they look at faces, and it sort of captures engagement,” he says. The research will be published in Nature Partner Journal’s Digital Medicine.

Several parents were stunned by changes, Voss says. “We refer to them as a ‘Light Switch Group,’ families that emailed us saying, ‘This is like a switch has been turned on [in their child] … what happened?’”

The Autism Glass Project gathers real-time data from the frames, processes the information in the smartphone and app including its artificial intelligence-based computing system, then beams information back to the user. Sensors that track expressions and eye movements are attached to the device to deliver real-time information about social cues to the user.

The device coaches users on eight emotions such as happiness, sadness, anger, fear and disgust, all based on psychologist Paul Ekman’s theory that these facial expressions show the basic human emotions. “Our theory was: if we can teach and give them confidence to [read these emotions], it would unlock their ability to learn more complex emotions,” says Voss.

Therapy that gives Hong Kong’s autistic children a fighting chance

The app they developed classifies the streamed images from the Google Glass, via a machine learning-based system, and beams information back to the user through the headset in the form of an emoji in a colour associated with the emotion. Audio feedback is attached to the headgear so the user hears a robotic voice saying, “Happy,” for example, if that facial expression has been recognised.

No known cure exists for autism. Symptoms are detectable in children as young as two years in the form of difficulties in learning, communication, and social interaction. A hallmark of this condition is difficulty in identify emotions.

According to Autism Children Foundation, there are about 25,000 kids with ASD in the city, and early intervention and therapy are vital. However, a perennial issue in Hong Kong is there are not enough therapists available and equivalent specialists from private practices usually charge HK$900 and up per session.

As a result, many parents join a waiting list for government-subsidised care, which can take from 12 to 24 months, according to the Rainbow Project’s website, an autism charity group in Hong Kong.

Autism group calls on Hong Kong government to offer more support services

The predicament is similar in China. Although figures vary, the prevalence of autism in China is estimated to be one per cent, or more than 13 million people.

“My guess is autism in China is probably under diagnosed, and even if you ramp up to get really solid diagnoses done to catch the kids early, you wouldn’t have the resources to take care of them,” remarks Voss.

“There aren’t enough therapists to take care of them so there is a crucial need to give tools to families they can use on their own while they are on a 12-month waiting list to see a therapist, or drive say five hours to see one,” he says.

Existing therapies include applied behaviour analysis that uses flash cards to coach autistic learners to identify emotions. Voss says such therapy is done in an isolated setting which makes it difficult for kids to translate insights in real-world settings.

“We’re trying to bring that therapy in the context where it matters, that start in the family home, where kids have their first social interaction,” he says.

His team is working on a licensing deal with an undisclosed company to get the device to market, hopefully in two years.

Asked about the cost – the original Google Glass had a US$1,500 price tag – Voss says ideally one would rent the device rather than own it, and the cost could be recovered through medical insurance. “Once you get an autism diagnosis, you should be eligible for guided therapy at home while you wait for therapy,” he says, adding that by the time you finally have access to a therapist, the autistic child would have made considerable progress.

Keith Lee, a project director at the Rainbow Project Learning Centre, says children with autism do respond well and sometimes better with visual aids than from reading words or listening to instructions. He cautions that the technology must be tested with a larger group with a range of degrees of autism, adding, “it’s hard to say if it will work for every kid, as everyone is different”.

Autism spectrum disorder covers a range of abilities, from those with severe learning difficulties and developmental delays, to high functioning children who can be integrated into mainstream schools. Lee says different approaches to learning apply based on their abilities.

Autism is a condition with complex and lifelong needs, he says, adding that it is hard to meet those needs as autistic children mature to adulthood. Many in Hong Kong have nowhere to go, so they stay at home, putting a lot of pressure on parents – especially those whose children display impulsive or self-harming behaviour and who may unintentionally hurt others.

“If they can find a place for their adult child to take part in meaningful activities, parents can have a break and have a better quality of life.”