Internet giant Yahoo! Inc. is adopting a new system for indexing Web pages that will charge businesses to include more material currently unlisted in its online search engine, marking the first volley in a duel with its former ally Google Inc.
Sunnyvale-based Yahoo is touting the approach, scheduled to be announced Tuesday, as a practical way to assure its search engine captures more of the so-called “Deep Web” — the billions of pages that aren’t found during periodic crawls of the Internet.
The method, often called “paid inclusion,” also will help Yahoo’s search engine keep better tabs on the most current material on a Web page, company officials said.
More than 99 percent of Yahoo’s search index will consist of Web links that don’t pay fees, said Tim Cadogan, the company’s vice president of search.
Search engine analysts generally applauded the move, saying it could open a rich new vein of content that’s lacking from all Internet search engines.
But the fees required to participate in the program are likely to raise worries about Yahoo creating an online caste system dividing the haves and have nots of the Internet.
To ease those concerns, Yahoo isn’t charging nonprofit Web sites to add unlisted links to its search engine. The nonprofit sites initially participating in the new indexing system include National Public Radio and the Library of Congress.
While Yahoo’s index will continue to include Web sites that don’t pay the fees, there’s no guarantee on how frequently those destinations will be visited nor how extensively the content will be analyzed, Cadogan said.
The fees won’t buy Web sites a higher ranking in Yahoo’s noncommercial search results, Cadogan said.
The fees under Yahoo’s “Content Acquisition Program” will be based on the size of the participating Web sites, how many unlisted links are submitted and how frequently the links are clicked on by the users of Yahoo’s search engine.
Google criticizes Yahoo move
Yahoo is counting on the program to give it an advantage over Google as it vies to supplant its rival as the Web’s most popular search engine. Yahoo licensed Google’s search engine for more than 3½ years, but started to cut ties with its former partner two weeks ago, vowing to introduce better ways to explore the Web.
Mountain View-based Google has built the Web’s largest search engine index, spanning 4.28 billion pages without charging fees to be included. Yahoo says its index contains “several billion” Web pages, but won’t provide specifics.
In an interview Monday, Google co-founder Larry Page called Yahoo’s new system “a pretty bad thing to do. There are plenty of profits to go around in search engines to find ways to improve the user experience without charging fees to do it.”
Like Google and other major search engines, Yahoo has long been using its search results page to display text-based ads that are tied to search requests. But these advertising listings are labeled as “sponsored results” and separated from the results generated through algorithmic formulas that are designed to provide an objective analysis.
Yahoo’s new system runs the risk of blurring the lines between the advertising and editorial sides of its search engine, said Chris Sherman, editor of Search Day, a newsletter published by SearchEngineWatch. “There’s definitely going to be a gray area,” Sherman said.
Forrester Research analyst Charlene Li said the distinctions might not matter to many Web surfers. “Most people don’t care whether something (in a search engine) is paid for or not. They just care whether its relevant. This could be an instance where people go to Google when they are looking for general information and go to Yahoo when they are shopping for something.”
News of Yahoo’s system rankled Gary Ruskin, executive director of Commercial Alert, a consumer group that that has criticized search engines for co-mingling their advertising and editorial results.
“The bottom line is that this is just going to be another way for businesses and the wealthy to buy search engine results so they get the material they want in front of the eyes of search engine users,” Ruskin said.