Helen: An AI-Powered Lipreading System for Accessibility and Enhanced Speech Recognition
Hearing impaired patients have a tough time communicating. However, existing hearing aids either fail to function efficaciously in noisy environments, or are too expensive for mass adoption. Helen, an AI Based Lip Reader, circumvents these issues by opening up a new dimension of quasi-audio-independent communication. Building on recent advancements in Visual and Audiovisual Speech Recognition, Helen uses a chain of neural networks to map sequences of lip movements to spoken words, and provides a transcription of spoken content even in non-ideal scenarios. While Helen was initially created as an accessibility wearable for the hearing impaired to supplement hearing aids, it became apparent that the system could also be used to substitute or supplement audio speech recognition in noisy environments, and enhance the usability of speech-to-text applications. Since debuting in April 2019, Helen has won multiple international awards including the Global Finals of the IET PATW in London, the James Dyson Award (National Runners up in Hong Kong), and the HKUST President’s Cup research and innovation challenge, among others, and its wearable form is currently patent pending.
Amrutavarsh Kinagi (Amrut) is an Undergraduate Researcher at the Hong Kong University of Science and Technology (HKUST). He has worked on creating systems that dramatically reduce cab-hailing time intervals by profiling city routines (which won his team hackUST, one of Asia’s largest hackathons), models that better determine driver reliability by analysing ride conversations during his internship at Uber and, more recently, worked with his batch mate, Padmanabhan Krishnamurthy, on developing Helen - an accessibility device for AI-Based Lip Reading. As the Head of the HKUST Intelligent Racing Robotics team, he possesses a keen interest in image processing, optimised path planning algorithms, and control systems, and has won awards in autonomous car racing events and autonomous humanoid control competitions. Amrutavarsh is currently pursuing a BSc in Computer Science and Mathematics, and a Minor in Humanities at HKUST.