On the reliability of identifying design moves in protocol analysis

This paper discusses issues and ways of measuring the reliability of segmenting verbal protocols of design activity, a central focus of design research. Reliability is an important issue in distinguishing ‘design moves’. In the present study, seven students working for a master in design degree, one graduated designer and two professors segment a 30 min protocol of a product design process into design moves. The intra and inter reliability was calculated for these observers using alpha coefficients. Neither the students', designer's nor professors' segmentation reached the desired cut-off value of 0.8. This negative finding questions the clarity of existing conceptions and urges more concise definitions, better training of analysts, and formulating more decisive instructions.

► When segmenting a protocol using non-syntactical rules, αU and dα are appropriate coefficients for measuring agreement.
► We identified the need of developing clearer definitions of design moves.
► We identified the need of developing more concise instructions for how they can be identified.
► We identified the need for further research regarding training analysts whenever non-syntactical are used.