“THESE search engines think they are private companies, but they’re not. How many times a day do you search using one of these engines? They’re everywhere, and it’s disingenuous of them to act as if they’re not public.” So says attorney Tim Hanigan of Lang, Hanigan & Carvalho in Los Angeles. Hanigan is representing Avi Datner, founder of party supply website Partypop.com, who last month filed a lawsuit against the search engine Yahoo. Datner claims that last year his website was suddenly removed from Yahoo’s search results. When he complained, the site allegedly reappeared for about a month and then mysteriously vanished again. Datner and Hanigan allege that Yahoo is manually altering search results to hurt vendors.
The company is not the first to file a lawsuit against one of the major search engines alleging they are unfairly skewing results by altering their rankings. Last month a US district judge dismissed a similar lawsuit against Google filed by KinderStart.com, which claimed the website had lost 80 per cent of its revenue after it plummeted down the search engine’s rankings. The company was given leave to amend and re-file its lawsuit.
Whatever the merits of the individual cases, there is a wider concern as to whether search engines’ algorithms really produce biased results. This matters, because search engines have become such an important source of information for us all – hence Hanigan’s claim that they should be treated as public organisations.
Advertisement
To a certain extent the search engines are victims of their own success. When they came onto the scene they were lauded as the ultimate guides to the internet: what until then had been a daunting ocean of information was suddenly more manageable, and their importance as a means of navigating the internet has grown ever since.
Almost as soon as the engines began operating, questions arose as to how they worked and whether they were biasing their search results. These questions remain. Helen Nissenbaum of New York University, who raised the idea of search engine bias in 2000 (IEEE Computer, vol 33, p 54), argues that because search engines have become such a pervasive technology, it is in the public’s interest to know how they work. The engines have operated for too long without any accountability, she says, and should not be allowed to keep their search functions secret as this could hide damaging social or economic bias. “Are there socially problematic biases built into search engines? We don’t know, because we don’t know how they work.”
Nissenbaum now hopes to find out. “These lawsuits are important because they can start the discussion of search engine transparency. Just having them out there, people will start saying, ‘Wait, wait, what’s going on here?'”
Studies since 2000 have largely supported Nissenbaum’s notion that search engines are inherently biased in the way they return results. Most researchers have argued that this is because search engines and the hyperlinked structure of the internet operate in a “rich-get-richer” fashion: popular sites become more popular, while smaller and newer sites are marginalised. Google’s PageRank algorithm, which forms a significant part of the engine’s larger search algorithm, certainly seems to work in this way. To index a website, PageRank counts the number of links to that site as votes in its favour. It then rates each link based on the popularity of the page it is found in. More links from more popular pages result in a higher ranking for the site in question.
Since users click mainly on pages that rank high on a list of results, researchers argued that popular pages would be made more popular and receive ever increasing traffic. Less popular sites would stay low in the rankings. The idea of this “Googlearchy” has persisted for years, but evidence is emerging that it may be flawed.
In a study published last week, researchers from Indiana University found that while the internet does indeed tend to make popular sites more popular, search engines mitigate this effect (Proceedings of the National Academy of Sciences, DOI: 10.1073/pnas.0605525103). To test the rich-get-richer notion, the team created three models of the internet to see how they affected traffic to popular and less popular sites. In the first simulation they ran searches using just the PageRank algorithm and assumed that user queries were random. In the second, they removed search engines from the internet, only allowing for page-to-page browsing via links. In the third model they used 4.5 million real-life user queries from AltaVista.
What they found was that when search engines scan using only PageRank, bias is high and popular sites stay popular, while less popular sites are overlooked. In a world without search engines a bias toward popular pages still exists, but is less pronounced. In the third model, which employs real user data, they found that bias is significantly reduced.
“It turns out that the queries people submit are incredibly heterogeneous, and very specific,” says Filippo Menczer, one of the authors of the study. “There is a bias inherent in the internet,” Menczer says, “but search engines mitigate that bias rather than amplify it. It doesn’t go away entirely, but you can see it’s more spread out.” This effect ensures that smaller, less popular sites can find their way to the top of a search engine’s list of results from certain searches.
“Almost from the start, questions arose as to whether search engines were biasing results” were biasing their search results would be made more popular”
Peter Norvig, director of research at Google, says that before search engines were created, and hyperlinks were the only way in which a website could be found, those to which there were few or no links were likely to remain unseen forever. That is no longer the case, he says. “That’s really the point of what we do. If you’re looking for a needle in a haystack, we’ll find it.”
If search engines do indeed offset a bias inherent in the internet, lawsuits such as Datner’s case against Yahoo could actually represent a threat to that balance, says Eric Goldman, who researches cyber-law at Santa Clara University in California. Goldman argues that attempts to expose the inner workings of search engines – supposedly for the benefit of the public – will in fact reduce their usefulness to ordinary internet users. If everyone knew exactly how the engines worked, organisations or individuals could manipulate them for their own ends, making them even more biased, he says.
Eight years after Google’s founders Sergei Brin and Larry Page published their paper detailing the algorithm that would be developed into PageRank, search engines are more widely used than ever. “If these lawsuits can force search engines to change, it could have a significant social impact that would do us all a disservice,” Goldman says.
Opening pandora’s box
Should any lawsuit over website ranking survive the initial hearing and reach the pre-trial stage, it could be commercially disastrous for the search engine concerned.
During the pre-trial information-gathering process the plaintiff could gain access to the search engines’ most closely guarded trade secret – their search ranking algorithm. And given the uncertainties of internet law in the US, they have every reason to be nervous.
Take the issue of “use in commerce”, which defines when use of a trademark constitutes an infringement, and when it does not. Search engines are in the lucrative habit of selling “keywords” such as company names to vendors to allow them to advertise their sites. When a user searches that keyword, the site comes up as a “sponsored link”.
The problems start when one company buys keywords found in another company’s name. The courts appear to be undecided over whether this is legal or not, and have ruled inconsistently in recent cases. “In some ways, the law applicable to search engines selling keywords has not advanced in any meaningful way since 2000,” says Eric Goldman of Santa Clara University in California.
For example, in March real-estate agent Edina Realty sued competitor TheMLSOnline for trademark infringement after it bought keywords such as “Edina Realty” from Yahoo and Google to direct searches to its sponsored link. The court found there were grounds for a trial.
Just 10 days later, a similar lawsuit filed by pharmaceutical firm Merck of Whitehouse station, new Jersey, against canadian internet pharmacy Mediplan global health was dismissed. This has left search engines and website owners none the wiser as to whether the practice falls foul of the law.
