Allowing computers to understand and translate sign language has long been a goal of developers aiming to empower deaf or hard of hearing people. Microsoft's Kinect depth-sensing camera has allowed major strides in this area — as demonstrated by this Microsoft Research project.
The team, composed of members from Microsoft Research Asia and the Chinese Academy of Sciences, presented their work at Microsoft's 2013 Faculty Summit earlier this week.
Kinect can detect where your limbs are as well as, to a certain extend, the position of your hands and fingers. By tracking the position of the user's hands closely, many signs can be detected instantly. The video above shows the Kinect's "thought process."
Of course, one might ask, when someone who is deaf is using a computer, why they don't simply use the keyboard? They do, of course, but sign language is the primary means of communication for many, just as spoken language is for those who can hear. Interacting naturally with computers and phones through speech recognition is something consumers are interested in, and training computers to recognize gestural language seems a natural extension of that.
There have been other projects to digitize sign language, but this is perhaps the most promising; electronic gloves have good performance but are expensive to make, and while ordinary webcams might work for some gestures, they're far from ideal due to differences between cameras.
On the other hand, the Kinect is widely available, reasonably accurate, and the software behind this can likely be adapted to other depth-sensing cameras — perhaps even Microsoft's own improved Kinect, which will be available in November with the Xbox One.
The software isn't ready for prime time yet — it's still just an experiment, albeit a successful one. But perhaps it won't be long before the hearing impaired can sign to the computer just as easily as they might to each other. You can read the paper the team presented this week at this link (PDF).
Devin Coldewey is a contributing writer for NBC News Digital. His personal website is coldewey.cc.