Observer reliability of industrial activity analysis based on video recordings

The aim of this study was to assess the agreement between observers analyzing activity patterns during truck engine assembly work based on video recordings. Two observers observed the recordings of nine workers, on the average 2.2 h long, assigning activities to four activity categories. For each activity category data were obtained on the mean duration of uninterrupted sequences of activities and their relative time proportion in the job. This data was analyzed with 2-way crossed ANOVA algorithms to derive the components of variance attributed to disagreement between observers, to differences between filmed subjects, and to residual “unexplained” variance. The latter was interpreted as an estimate of within-observer variability and possible interactions between subject and observer. While the observers disagreed about the overall time proportions for the four activity categories by no more than 3.7% of time, their second-to-second classification disagreed for 13% of the total analysis time. The between-observer variance was small as compared to within-observer variance and the variance between subjects performing the same job. Simulations based on the variance components showed that a group mean of the proportion of direct work could be determined with a standard deviation within 5% of the mean by having two observers analyzing one 2-h video recording once, each.Relevance to industryThe results of this study may support decision making when designing a reliable video-based analysis of industrial work. Thus, the study helps production engineers, ergonomics practitioners and researchers allocate resources between data collection and data analysis, based on their preferences for precision and power of a particular study.