嶺井第一病院 in Urasoe City, Okinawa Prefecture, Japan, has begun providing the CommunicationBoard+AI system to help patients who have lost the ability to speak while retaining awareness and the ability to express themselves. The system relies on MiRZA glasses from NTTコノキューデバイス, a smartphone paired with them, and an XBOX Adaptive ジョイスティック controller.
During a trial that ran from January through April, 25 patients participated cumulatively. According to the article, some patients were hesitant at first, but continued using the glasses enthusiastically after trying them. Composing messages, which had taken minutes using a paper or acrylic letter board, took tens of seconds or, in some cases, a few seconds. The system was officially made available on May 29, and individual use has also begun among patients with amyotrophic lateral sclerosis (ALS).
From a Letter Board to Sentence Suggestions
A Japanese syllabary board made up of “gojūon” syllables appears within the user's field of view. The cursor can be moved through head movements or with the controller, with two input options: selecting each character individually, or selecting a group such as the “A row” or “Ka row” and then narrowing the selection in a manner resembling a sliding keyboard. Staff determine the most suitable method for each patient.
After approximately two characters have been entered, the system uses an OpenAI artificial intelligence model available through Microsoft Azure to predict the next words or phrases. Once the sentence is approved, it is converted into speech using the Text to Speech service so that family members or care workers can hear it.
The Camera Adds Context to the Message
The system is not limited to text input. When the camera icon is selected, Azure's image-recognition service analyzes the scene captured by the glasses' camera and suggests the names of objects such as curtains, a bed, and a television. The message can then be completed with verbs such as “turn on” or “turn off.” In other examples, the system identified books, fruit, and people appearing in front of the camera, then added appropriate names to the suggestion list.
Rocket Studio developed the system over three and a half years, beginning in November 2022, with a team consisting of a programmer and a manager. The company's vice president, Tomohiko Nakajima, said that the core innovation does not lie in artificial intelligence technology developed by the company, but in interface design: colors, character sizes, the number of input steps, and the ability to use it without reading a manual. The company drew on its experience in game development to build the interface.
What Changed in Practice?
The system's value lies in reducing the burden between a patient's desire to express something and the message reaching the people around them. In addition, placing the artificial intelligence and speech-conversion services in the cloud made it easier to transfer the application from HoloLens 2, whose production stopped in 2024, to MiRZA without rebuilding the core components; the main modifications were limited to input and display on the device.
Field experience also showed that head-movement control can cause fatigue, particularly for some rehabilitation patients in wheelchairs. Therefore, the company added the controller after the hospital's observation and managed to integrate it within one day. Each patient's experience is customized through preset text instructions, such as presenting phrases related to lighting or curtains for a patient whose eyes should not be stimulated. The system does not load the entire electronic medical record, but is limited to the necessary information, according to the article.
Limitations and Open Questions
The current control method does not work for an ALS patient who can no longer move anything except their eyes; it requires use of the head or finger, meaning that use should begin before these abilities are lost. Rocket Studio is considering transferring the system to less expensive devices equipped with eye tracking if they become available, and is also considering adding an artificial voice that imitates the patient's original voice by recording phrases in advance.
The service is sold through licensing agreements, while the price is determined individually upon inquiry, and the company plans a lower-cost version for home rehabilitation. The article mentions an automatic filter for harmful content in Azure, but provides no quantitative clinical results or operational details regarding the accuracy of word prediction and image recognition—points that require review before assessing the system's suitability on a broader scale.