An evaluation dataset for the toponym resolution task

Toponym resolution is the task of linking place name instances in a text with spatial footprints, given the context in which they occur. Whereas a lot of work on the evaluation of temporal resolution is ongoing (e.g. [Setzer, A., & Gaizauskas, R. (2000). On the importance of annotating temporal event–event relations in text. In LREC 2000 Workshop on annotation standards for temporal information in natural language, Vol. 3 (pp. 1281–1286). Athens, Greece]), to date no reference resource is available to evaluate competing algorithms for toponym resolution. It is thus argued that a shareable, reusable evaluation resource is necessary.To this end, a new proposal for the markup of toponyms in text corpora with their referents and an associated tool data methodology are presented: the Toponym Resolution Markup Language (TRML) is an XML-based markup language, and TAME, the toponym annotation markup editor, is a tool that implements it. A novel evaluation resource is described which comprises a large-scale reference gazetteer server and a human-annotated news corpus in which toponyms are associated with latitude/longitude coordinates of the location they refer to. The reliability of the annotation task is established by determining inter-annotator agreement of the human annotators.