What makes an item more difficult? Effects of modality and type of visual information in a computer-based assessment of scientific inquiry abilities

• We designed a multimedia-based assessment of science inquiry abilities.
• We examined the effects of modality and visual information on test performances.
• Overall items became slightly more difficult when the item presentation was static.
• The interaction effect between modality and grade was significant.
• The modality effect varied when content visuals were used.

Effects of multiple external representations on teaching and learning have been widely researched; however, relatively little is understood about how different types of representations used for item presentation influence students' performances in computer-based assessments, particularly in those evaluating complex abilities. We designed a multimedia-based assessment for secondary school students that focused on scientific inquiry abilities (i.e., questioning, experimenting, analyzing, and explaining). Through the participation of 1218 students (561 8th graders and 657 11th graders), the balanced arrangement of test booklets, and the use of a generalized partial credit Rasch model, this study aimed at investigating how the modality of representations for item presentation (i.e., dynamic and static) and type of visual information (i.e., context and content) conveyed by the representations affected the item difficulty of the multimedia-based assessment. The results showed that overall items became slightly more difficult for students when the item presentation was static. Also, the interaction effect between modality and grade was significant; when item presentation was changed from dynamic to static, the items became easier to the 8th graders, whereas they were more difficult to the 11th graders. The results suggested that older students might have more cognitive resources to retrieve information from dynamic displays to solve the assessment tasks successfully. Additionally, while all interactions between modality and type of information on items with context visuals were not significant, there were significant interactions on four items with content visuals. A further examination of the items suggested that the tasks and the phenomena involved in the items may influence how modality affected the item difficulties when content visuals were used. These results implied that researchers need to pay special attention on the design of content visuals because they appear to be more influential than context ones to students' performances of science assessments.