Assessing reproducibility in faunal analysis using blind tests: A case study from northwestern North America

Zooarchaeologists have long recognized that assigning taxonomic identifications to animal remains is a subjective process, and recent studies have highlighted the need for data quality assurance standards in archaeofaunal research. Our study contributes to this growing interest in quality assurance by presenting simple quantitative methods for assessing the reproducibility of analytic results through blind reanalysis of animal remains that we developed during analysis of fishbone from Čḯxwicən, a large Native American village on the coast of Washington State, U.S.A. Given the large scale of the Čḯxwicən project - over 112,000 fish remains were documented by five different analysts over three years - there was a real possibility that inconsistencies in laboratory practices affected analytic results (e.g., number of identified specimens, taxa present and relative abundance, elements identified, burning frequencies). To evaluate the reproducibility of the Čḯxwicən fishbone data, and the possibility of “protocol drift” - changes in how specimens and bone modifications were identified over the course of analysis - we reanalyzed samples of fish remains that were previously documented during three discrete stages (beginning, middle, and end) of the Čḯxwicən project. The original data and the reanalysis results show close agreement in each stage, with only minor differences in the numbers of recorded specimens, taxonomic representation at family- and finer taxonomic levels, and body part representation assessed for a single taxonomic order. Identifying burning on bone was not very reproducible. Reproducibility studies such as this are useful for highlighting ambiguous identification criteria (e.g., in taxonomic assignment, bone modification), and could stimulate dialog among researchers about ways to address such issues in future studies. Increasing the implementation of these, and other, widely applicable methods should improve zooarchaeological data quality and stimulate further research on quality assurance in archaeology overall.