Using a game controller to interact with real-world objects is definitely spooky. You push around a glorified pencil to "feel" the contours of a hand resting on a faraway table. And if that faraway hand moves, you'll feel an unseen force push back. It's as if an occult hand were taking control of the magic pencil from yards or miles away.
The push of a ghostly hand, vs. the virtual sense of touch ... it's not easy for me to say which aspect of the University of Washington's Kinect-based robo-control system is spookier. But it's easy for Fredrik Ryden to say which aspect is more useful.
"We want to give robotic surgeons a sense of touch," the visiting graduate student from Sweden told me.
The point of Ryden's contraption is not merely to manipulate objects over far distances. Heck, even a monkey can use a thought-controlled robotic arm to pick up distant objects, and surgeons have been operating remote-controlled robotic scalpels for years. But it takes a more sophisticated kind of robot to give those surgeons tactile feedback about how deep they're cutting, and create a virtual force field to keep their scalpels from straying.
The fact that Ryden's robo-touch system could demonstrate that capability after just a weekend's worth of work, using a $150 motion-sensing game device, adds to the experiment's geek appeal.
"I realized what I was doing was really cool, but it was easy — so I was surprised that nobody else had done it," Ryden said.
Now that the feat has been publicized on YouTube, in the blogosphere and beyond, it seems as if everyone is trying to do it, said Howard Jay Chizeck, an engineering professor who's co-director of the University of Washington's Biorobotics Laboratory. "The sense I have is that we're just a little bit ahead of whoever is right behind us," he joked.
How it works
Microsoft (which is a partner in the msnbc.com joint venture) sells the Kinect system as a "controller-less" controller for its XBox video game console. Players can interact with their games by gesturing, punching, jumping or even dancing in front of an infrared laser projector and a set of infrared depth sensors. Kinect's software analyzes the patterns of scattered infrared light to create a 3-D "cloud" of data points that reflect the players' changing positions in real time.
It didn't take long for computer geeks to hack into the Kinect system for a wide spectrum of unanticipated applications, ranging from "Air Guitar Hero" and a virtual-reality piano to extreme body jiggling and other risque pursuits. On the serious side, an outfit called Virtopsy has programmed Kinect to serve as a touch-free interface for medical imagery in operating-room environments. And then there's the Biorobotics Laboratory's hack.
Under the direction of UW's Blake Hannaford, the lab has been working for years to develop better robotic surgeons for military as well as civilian use. Surgical robots are already widely used for delicate operations such as prostate removal, but medical experts in the military (and at NASA) would love to have robots that can do a wide range of surgical operations by remote control, from hundreds of miles away.
So the Biorobotics Lab was challenged to come up with a system that could provide real-time feedback to the surgeons at the robot's controls — including a way to warn the surgeons if they were getting too close to a vital artery or some other danger zone.
Think of it as a 21st-century, virtual-reality "Operation" game with real-world consequences. Bzzzzt!
"Essentially, you're projecting a sense of touch through an image," Chizeck explained. "We'd like to have images of things generate 'force fields' around things you don't want to touch."
When the Kinect system came out in November, Ryden saw it as the perfect platform for such a device. His software translates the cloud of data points into a virtual 3-D surface. When the magic pencil (actually, a software-controlled stylus at the end of a robotic arm) "hits" the virtual surface, it moves no farther — just as if it were hitting the real surface of a faraway hand. The same thing can happen if your stylus strays up to the edge of the force field. (Though actually, if you press hard enough, you can push the stylus through the force field. It feels as if you're poking a pin through a piece of virtual cardboard.)
What it's for
The robo-touch system is currently being fine-tuned as part of the Biorobotics Lab's long-running project on surgical robotics. The beauty part is that buying Kinect systems doesn't strain the lab's hardware budget, Chizeck said. "It's 150 bucks for a system that would cost maybe $100,000 or $150,000 to reproduce," he said.
That doesn't mean low-cost Kinects will be showing up in operating rooms. The low-cost devices don't have anywhere near the resolution that the actual robo-touch device would require. Eventually, super-sensitive touch feedback systems will be built from the ground up and put through clinical trials, as part of a long and potentially expensive development process.
"You're talking at least a decade," Chizeck said. "I think in Fredrik's lifetime, it's a sure bet, but it's really hard to predict." (Fredrik Ryden is 22 years old.)
Hannaford told me that surgical robotics may turn out to be the "killer app" for the field of haptics, which focuses on methods for translating virtual-reality shapes into a real-world sense of touch. (Maybe "killer app" isn't the best phrase to use when talking about medical procedures, but you get the point.)
Chizeck had a slightly different take: "I'll make a bet with Blake," he said. "I think there'll be a game application using haptics before there's a patient operated on."
He said robo-touch technology could also be used to create more dexterous bomb-disposal robots and deep-sea autonomous vehicles. But there's one obvious application that no one in the lab was willing to discuss: the use of haptics for long-distance, virtual-reality sex.
"I'll let you think of your own apps," Chizeck said.
More on gaming and virtual reality:
- Virtual haven set up for combat vets
- 'Star Wars' holograms nearly a reality
- Virtual actor takes over in 'Tron'
- Kinect hacks unleash your inner superhero
Join the Cosmic Log community by clicking the "like" button on our Facebook page or by following msnbc.com science editor Alan Boyle as b0yle on Twitter. To learn more about Alan Boyle's book on Pluto and the search for planets, check out the website for "The Case for Pluto."