If a picture’s worth a thousand words, what can you say with two million snapshots?
With some help from the Flickr photo-sharing Web site, two researchers at Pittsburgh’s Carnegie Mellon University have shown how a new picture-patching program can transform flawed vacation shots into “Wow!”-worthy masterpieces.
Think of it as a bit of picture-perfect revisionist history for the digital age. Or how those snapshot souvenirs from Europe might have looked if that lovely bay hadn’t been blocked by a roof or the charming plaza hadn’t been marred by your two-timing ex.
Unlike existing programs that use bits of the same photo to patch holes, the new program relies on an algorithm that first searches through heaps of digital photos — 2.3 million downloaded from Flickr in this case — for ones that match the gist of the scene. The sophisticated formula tries to match general properties such as shapes, textures and orientations to pick out all photos with, say, a similarly curving bay or a river running through a city.
The program then looks within that subset for good patches by blending candidates with the target photo and finding boundaries that would be least noticeable to viewers. In the finished product, that view-blocking roof might be replaced with sailboats or your ex-boyfriend supplanted with some greenery.
Although attempting to gather all possible images of the world would be pointless, the researchers say their study suggests it might be feasible to collect enough representative scenes so that any input image would yield a “similar enough” photo to produce a realistic composite.
Graduate student James Hays and assistant robotics and computer science professor Alexei Efros are careful to point out that their digital patch is not intended to accurately restore all the information that should have been there but to fill in the missing pixels with images that could have been there. Or at least look like they belong.
Efros said getting a computer to produce a composite scene that is not only seamless but also contextually valid reflects a main challenge of artificial intelligence research.
“If you have a cow standing in a field and you erase the cow, it’s not hard to use the green grass in the field to fill in the hole,” Efros said. But removing a car parked in an alleyway and convincingly filling in the image, he said, is a far bigger hurdle.
“The computer needs to realize, ‘Ah, the car is gone but what’s usually underneath the car is a road, so let’s fill it in,’” Efros said.
Humans are particularly good at using visual cues for such problem-solving, he said, but the field of artificial intelligence is “nowhere near” a solution that allows computers to likewise interpret new scenes as an alleyway or beach or airport check-in counter.
One of the new study’s chief insights is that the problem might be effectively bypassed with a big enough database. With 10,000 images, Hays and Efros found that their program’s closest matches weren’t all that similar to the target photos. With more than two million images, though, they found convincing patches for a significant percentage of their incomplete scenes. For future research, they hope to amass 10 million pictures.
In a way, Efros said, “we’re cheating the artificial intelligence problem by doing this Google-style lookup.”
Perhaps unsurprisingly, Google has expressed early interest in the approach. The study, presented by Hays earlier this month in San Diego at the Association for Computer Machinery’s International Conference on Computer Graphics and Interactive Techniques, also suggests a method for how the technology and future iterations might be evaluated.
With unlimited time to scrutinize the photos, a group of study participants thought 37 percent of Hays and Efros’ doctored images were genuine, but were fooled by only one in 10 photos altered by a program that uses bits of the same picture as a patch. Untouched photos, by contrast, were labeled as the real deal 87 percent of the time.
Antonio Torralba, an assistant professor in the Computer Science and Artificial Intelligence Laboratory at Massachusetts Institute of Technology in Cambridge, called the new research a “very exciting” demonstration of what can be accomplished with such large datasets.
He should know. For his own research, Torralba has amassed a collection of 80 million low-resolution images, each just 32 by 32 pixels. Even at that low resolution, he said, a computer program can pick out enough general details like orientation and texture to reliably retrieve most images that contain, say, a dog.
For their photo-patch, Hays and Efros incorporated Torralba’s “gist descriptor” code into a larger program that sifted through the Flickr database. Sometimes, the search results suggested that a mountain stream could take the place of a similarly oriented street running toward a historic building. And occasionally, the patch even yielded a geographically precise replacement (like how a basilica would really look without its scaffolding). Ultimately, though, Hays and Efros said the photo’s final appearance would be up to each individual user.
Such blemish-obscuring tools may seem like the equivalent of testosterone shots among Tour de France cyclists. But, as Torralba noted, tinkering with celebrity photos to add scandalous liaisons or subtract unsightly blemishes is nothing new.
“Faking pictures is something that has probably happened since the beginning of photography,” he said. “Of course, this probably provides you with a tool to make it easier.”
Even with 2.3 million images to call upon, however, the process is far from perfect. If erasing a persona non grata also removes the leg of a bystander, for example, similar scenes are unlikely to offer a realistic reconstruction of the missing limb. Far more nettlesome may be the new uncertainties the technology poses for copyright protection laws. Does borrowing a piece of one image to patch another, for example, fall within the fair-use exemption under an image’s copyright protection?
If the outstanding technical and legal issues can be resolved, Hays and Efros ultimately envision a Web-based service in which consumers would be able to submit an incomplete photo and get back a range of possible patches.
In principle, Torralba said, such a program has a promising commercial future. And for someone determined to digitally scrub all traces of a former beau from the canals of Venice, the technology could well be priceless.