Search markets and search results: The case of Bing

• Webometric studies should combine results from three or more search markets.
• Search market choice altered a small majority of the Bing top 10 results.
• Search market choice altered less than a third of the complete sets of results.
• Variation in the Bing top 10 results over time was greater for complete results sets.
• Searches returned almost no ubiquitous authoritative results (individual URLs).

Bing and Google customize their results to target people with different geographic locations and languages but, despite the importance of search engines for web users and webometric research, the extent and nature of these differences are unknown. This study compares the results of seventeen random queries submitted automatically to Bing for thirteen different English geographic search markets at monthly intervals. Search market choice alters a small majority of the top 10 results but less than a third of the complete sets of results. Variation in the top 10 results over a month was about the same as variation between search markets but variation over time was greater for the complete results sets. Most worryingly for users, there were almost no ubiquitous authoritative results: only one URL was always returned in the top 10 for all search markets and points in time, and Wikipedia was almost completely absent from the most common top 10 results. Most importantly for webometrics, results from at least three different search markets should be combined to give more reliable and comprehensive results, even for queries that return fewer than the maximum number of URLs.