A wireless system consisting of seven rings worn on the fingers can translate sign language in real time. Sensors measure the position and movements of the fingers and forward them to an AI system, which reconstructs words and sentences from them. So far, the system has been trained on a hundred words each from American and International Sign Language, which it recognizes with an accuracy of almost 90 percent. From the development team’s perspective, wearable technology could allow deaf people to more easily communicate with people who don’t understand sign language.
In sign language, hand gestures, body movements and facial expressions convey meanings that are then combined into sentences. For many people with hearing or speech impairments, sign language is the primary means of communication. “However, people without prior knowledge of sign language are usually unable to understand or respond to it, which leads to a constant communication barrier in everyday interactions with sign speakers,” explains a team led by Jaejin Park from Yonsei University in Seoul, South Korea. “In addition, more than 300 different sign languages are used worldwide, each of which has developed from regional and cultural contexts. These differences make it difficult for sign speakers from different regions to communicate with one another.”
Translation devices are intended to make communication easier for sign speakers. Previous systems are usually based either on cameras that record movements or on sensors worn on the hands, often integrated into gloves. However, these devices are often not suitable for everyday use – for example because the cameras only work reliably under certain lighting conditions, the gloves have to be laboriously individually adjusted or the sensor cables restrict freedom of movement.
Sensors on seven fingers
Now Park and his colleagues have developed a translation system that is intended to overcome previous limitations. Your sign language translator consists of seven rings that are worn on the fingers and use sensors to measure the acceleration of the fingers and their orientation in relation to gravity. In this way, the rings recognize both dynamic and static gestures. Using Bluetooth, each ring transmits the signals wirelessly to a device that brings together the information, evaluates it with the help of artificial intelligence and translates it into words and sentences.
During the development of the device, it became apparent that the right thumb conveyed the most information. Additional rings record the movements of the index, middle and ring fingers of the right hand as well as the index, ring and little fingers of the left hand. “Thanks to the independent, wireless sensor rings, our system enables full finger mobility,” explain the researchers. Unlike previous devices, it does not require customization. In tests, hand gesture recognition also worked reliably for people who had never used the device before.
Real-time communication
The system was trained on one hundred words each of American and International Sign Language. It recognized these in tests with an accuracy of almost 90 percent. It was also able to reconstruct sentences from a sequence of the known words without the need for separate training on sentence structures. “The device thus paves the way for fluid sign language communication in real time,” write Park and his colleagues.
Although the AI rings only record finger gestures and ignore other factors such as facial expressions, the researchers believe they could help break down barriers to communication with sign language. “They could enable seamless interaction between sign speakers and non-sign speakers in a variety of practical contexts,” the team said. The technology could also be used outside of sign language, for example to control virtual reality applications.
Source: Jaejin Park (Yonsei University, Seoul, South Korea) et al., Science Advances, doi: 10.1126/sciadv.aec8995