Researchers at the University of Texas, Arlington, are developing technology that would allow users of virtual reality (VR) headsets to see the real facial expressions of the people they are interacting with.

“This is one of the first devices that will allow us to monitor human facial activity in detail and it has a variety of potential applications ranging from [VR] gaming to healthcare,” said Professor VP Nguyen, a computer scientist at Texas.

“The project bridges the gap between anatomical and muscular knowledge of the human face, and electrical and computational modelling techniques to develop analytical models, hardware and software libraries for sensing face-based physiological signals.”

Nguyen is director of the university’s wireless and sensor systems laboratory, which is focused on building connected systems to monitor and improve human health and environment. He has received a grant of almost $250,000 from the National Science Foundation for the VR project, which is one part of a larger grant with collaborators from the University of Tennessee-Knoxville.

Current VR headsets remain bulky - despite considerable R&D effort from companies including Facebook to reduce their size, whilst retaining immersion and a crisp image - and block the majority of the user’s face. Nguyen’s team has created a device which it describes as lightweight, unobtrusive and privacy friendly.

“Monitoring human facial muscle activities at fine-grained details and reconstructing 3D facial expressions are challenging tasks,” said Professor Hong Jiang, who is chair of the university’s computer science and engineering department. “This project will advance the state-of-the-art ear-worn sensing techniques and transfer what is learned across multiple sensing modalities.

“The findings from this research could enable a wide range of applications, ranging from virtual and augmented reality to emotion recognition in healthcare.”

The scientists hope the device could, specifically, have applications in speech therapy. For instance, physicians and therapists could use them to work with remote patients who have had strokes or other speech impediments, pinpointing exactly where the muscles are affected and tailoring treatment around that information. Additionally, it could be used along with phones in noisy environments such as subways or bars, where speech-recognition technology cannot perform well.

There is hope among healthcare professionals that extended-reality technologies such as VR and AR could have considerable benefits for patients with mental illnesses, neurological conditions and developmental disorders. For instance, a mobile VR game could provide warning signs of dementia, while a controlled, immersive environment called ‘The Blue Room’ has helped children with autism overcome some phobias exacerbated by the condition.