The risk of using the Internet as reference resource: A comparative study

ObjectiveWe sought to examine the frequency and the specific problems encountered in accessing Internet references in two leading medical journals during the last 3 years.MethodsTwo investigators independently reviewed all publications in the issues of the New England Journal of Medicine and The Lancet during October 2005 to March 2006, November 2004 to January 2005, and November 2003 to January 2004. We calculated the total number of references and the subset referred to an Internet source of each article. Then, we visited the electronic sources to identify the Internet references and noted the problems of accessibility, if any. When we failed to directly access the reference in the electronic address provided by the authors, we visited the referred website; if this was also inadequate, we performed Google searches to retrieve the missing reference(s).Results465/18,850 (2.5%) and 952/24,630 (3.9%) of the reviewed references in the New England Journal of Medicine and The Lancet, respectively, referred to Internet sources; from these we could not access 68/465 (14.6%) and 170/952 (17.9%) in the two journals, respectively. There were increasing proportions of lost Internet references as they age. Searching into the website referred by the authors of the reviewed articles could not provide the missing information in a considerable proportion (62.2%). However, the use of an Internet search engine (Google) helped us to identify references in other websites, reducing the proportion of missing Internet references to 17/465 (3.7%) and 17/952 (1.8%) for the two journals, respectively.ConclusionsThe response “page not found” was commonly encountered when we tried to access Internet references in publications of leading medical journals during the last 3 years. A considerable proportion of missing references was identified with the use of Google search engine. Authors of scientific articles should be aware of the problem of missing Internet references and until well-established Internet archiving solutions are in use, they should choose carefully their Internet references from reliable websites whenever it is impossible to avoid using them.