Using scantron versus an audience response system for survey research: Does methodology matter when measuring computer-mediated communication competence?

The purpose of this study was to investigate whether results of the computer-mediated communication (CMC) competency scale differ depending on the survey instrument used. Data were collected from 173 undergraduate students (53% female; average age 20). Of these, 96 subjects completed the questionnaire using scantron sheets, while 77 subjects used an audience response system called Classroom Performance System (CPS). All subjects completed a second questionnaire using paper-and-pencil to evaluate their experience with either the scantron sheets or the CPS. t-Test results show that using the CPS is more fun for participants (p<.000); no more or less difficult (p=.806); and does not make a difference with regard to experienced time pressure (p=.425). Subjects have higher doubts about the validity of their recorded answers (p=.007) when using the CPS. Regression analysis shows that in two out of 12 constructs of the CMC competency scale, scoring was influenced by using a technological data collection method. The two constructs were “message factors” (p=.002) and “medium factors” (p=.002). Overall, results support the use of the CPS as an alternative to scantron sheets for measuring CMC competence.