China study turns brain activity into Mandarin, breaks language barrier
- Researchers adapt algorithms to recognise tones, giving hope to Chinese-speaking patients with communication disorders
- The model outperforms existing methods to produce mostly clear and recognisable synthesised speech
The team, led by Wu Jinsong from Huashan Hospital – affiliated with Fudan University’s Shanghai Medical College – said the advancement could help to restore speech capabilities in communication disorder patients who speak tonal languages.
Advances in brain-computer interface technology have already led to machines that can articulate brief phrases in English and Japanese from recordings of brain activity. But the unique challenges of Mandarin have so far been beyond their capability.
The study, in collaboration with researchers from Tianjin University and ShanghaiTech University, plugs a research gap with a method to decode and articulate Mandarin speech from brain activity, the scientists said.
The neurolinguistic structure and pronunciation of Mandarin meant that English-based neural mechanisms and algorithms could not be directly adapted. Syllables with the same structure but different tones can represent various words.
For instance, the Mandarin syllable “ma” has four different tones that can mean “mother”, “hemp”, “horse”, and “scold”, respectively. To navigate this complexity, Wu and his team enhanced the algorithms that observe neural activities.