TREC chemical information retrieval – An initial evaluation effort for chemical IR systems

There is little need to emphasize the importance of chemoinformatics and chemical information retrieval. However, what seems to require a lot more effort in convincing members of the community is the need for standardized evaluation procedures and measures. How confident are we, as users, that the search tools we used have given us all the information that we were looking for? It is unrealistic to believe a marketing campaign which claims that a specific patent IR (information retrieval) system can find all relevant results for a search topic. And if we don’t trust marketing campaigns, how can we get an idea of what to expect from a patent search engine?One of the most prominent initiatives to work in this direction, of evaluating chemical IR tools, has started in 2009 with the support of NIST (National Institute for Standards and Technology): the TREC Chemical IR Track focuses on evaluation of search technologies for retrieval and knowledge discovery of digitally stored information on chemical patents and academic journal articles on chemistry. This paper describes our 2009 experience, presents the official results of the participating groups, and lays down the targets for 2010.