Have You Installed ACAP On Your Website – The Protocol That Can Control Yahoo And Google News Searches? No Problem, Neither Search Engine Is Using It Yet

Philip Stone December 20, 2007

Is the ability of Google News to search a news web site’s content and list that site among search results good or bad for newspapers? Is it good that Google can publish on its news site the first paragraph or so of a news item, fully credited to the referenced site and links the reader to that site? Much of the news media seems to believe all of that infringes upon copyright, limiting profits, and so they have come up with a new protocol that controls what the search engines can and cannot do.

The new system called ACAP (Automatic Content Access Protocol) is designed to let a web site block indexing of specific pages, or an entire site. It extends what was available from the Robots. txt command developed in 1994 to block content on a server and the meta robots that were developed to allow page-by-page blockage.

Newspapers who didn’t want Google picking up their copy, for copyright or other reasons, just had to apply the robots.txt language, but for reasons best known to themselves they opted instead to complain loudly that Google was violating their copyrights, getting news for free and so on.

Although it sounds a bit like semantics the case against robot.txt was that it was solely a blocking tool, whereas ACAP is considered a tool which communicates automatically with the search engines, telling the search engine robots what they can do with each page of copy – publish it entirely, publish only extracts, or not touch it at all. But if the search engines robots don’t “talk” to that tool, and so far they haven’t said if/when they will, then it’s not going to work.

The European media do have a copyright victory in Belgium they can use to pressure the search engines. In a celebrated court case, Copiepresse, the publishers association, sued Google for copyright infringement and the court ordered Google  to stop its practices in picking up news and ordered a €25,000 fine for each day it did not. Google has appealed, and the outcome is still awaited,  but the wheels of justice turn slowly, which gave a chance for a thaw in May between the two sides They agreed a system for Google to link the newspapers’ web sites, but accessing archived material is prevented by the newspapers using a “no archive” tag to stop the Google robot when it came across an item the publishers didn’t want cached. 

The European Union in the form of Viviane Reding,  European Commissioner responsible for Information Society and Media, strongly endorses ACAP. She said earlier this month she thought the process was a win-win for both sides “as publishers can link content with authorizations for access and use in a form that can easily be recognized and interpreted by a search engine crawler.”

The EU is particularly stern on copyright issues and she told a recent publisher’s forum “I very much hope that the companies offering search engines will cooperate with ACAP.”

At that same meeting Jean-Christophe Conti, vice-president of Yahoo Search Marketing, Europe, gave the basic search engine position that publishers should be wary for  “content is everywhere” so if the search engine doesn’t talk with an ACAP site then it can simply use those sites without ACAP. But Mrs. Reding also said the European Commission will shortly publish a strategy paper with the purpose of “starting a process aimed at facilitating business negotiations and improving legal certainty (translation: a paper to ensure the search engine companies fall into line).”

Google’s view, which in one form or another is the same as other search engines, is that  while it is always interested in initiatives that allow publishers and search engines to work closer together, and that work has been done to make the robots system more effective,  it must keep in mind that “as a broad-based search engine, we need to keep in mind the needs of millions of web publishers worldwide.”

ACAP has its detractors, both philosophically and technically. The philosophical objection is best summed up by Ian Douglas, head of digital production for the UK’s Daily Telegraph, who wrote, “Throughout ACAP’s documents I found no examples of clear benefits for readers of the websites or increased flexibility of uses for the content or help with making web searches more relevant.

“The new protocol focuses entirely on the desires of publishers, and only those publishers who fear what web users will do with the content if they don’t retain control over it at every point.

“The web, and electronic content in general, thrives when access is as free as it can be. Second-guessing what should be displayed in what circumstance, as ACAP attempts to do with its content thumbnails, is an attempt to read the future of how people will wish to search and the kinds of results they will respond best to.

“I want people to find Telegraph content in any way they choose. Be it through Google News, RSS, some obscure map mash-up I’ve never heard of (and need never become aware of), a link from a widget on someone else’s blog, I really don’t care. Come one, come all. The very idea of exclusion is ridiculous to any publisher with an advertising-based model that relies on traffic to pay the bills.

“Robots.txt does a fine job of excluding search engines from anything that might need to be hidden for legal reasons or to make sure we don’t present duplicate content to the world...

“The web is not print. You don’t get to say how people come to your content online, that’s just not how it works. In exchange for a little loss of control you gain a supremely flexible international audience who can interact with you every day. ACAP might well be adopted by a lot of publishers (although not, so far, by any search engines anyone has heard of), but we’ll all be a little poorer as a result.

Taking the opposite view is Gavin O’Reilly, president of the World Association of Newspapers, who said in a New York speech that “ACAP will give  the content industry worldwide the incentive to innovate, create and disseminate. Newspapers, magazines, books, journals, directory publishers, anyone involved in digital publishing can now adopt a standard that will protect their interests and will make them masters of their own content.” 

Interestingly, O’Reilley did not mention end users among those who will benefit from ACAP which rather supports Douglas’ view. The ACAP folks might want to take on that PR chore!

Also bloggers have been writing about technical flaws but since the protocol has just been launched no doubt there are fixes and improvements on the way. A common complaint, however, is that the software has not been subjected to round-robin peer review, something one might have expected from a major new protocol that is intended to replace the tried and tested robots.txt.

For those interested in testing and using the new protocol the-ACAP.org site site has helpful tools at no cost.