Do online readiness surveys do what they claim? Validity, reliability, and subsequent student enrollment decisions

Online readiness surveys are commonly administered to students who wish to enroll in online courses in college. However, there have been no well-controlled studies to confirm whether these instruments predict online outcomes specifically (as opposed to predicting course outcomes more generally). This study used a sample of 24,006 students to test the validity and reliability of an online readiness survey similar to those used in practice at a majority of U.S. colleges. Multilevel models were used to determine if it was a valid predictor of differential online versus face-to-face course outcomes while controlling for unobserved heterogeneity among courses taken by the same student. Student self-selection into online courses was also controlled using student-level covariates. The study also tested the extent to which survey score correlated with subsequent decisions to enroll in an online course. No aspect of the survey was a significant predictor of differential online versus face-to-face performance. In fact, student characteristics commonly collected by institutional research departments were better predictors of differential online versus face-to-face course outcomes than the survey. Furthermore, survey score was inversely related to subsequent online enrollment rates, suggesting that the use of online readiness surveys may discourage some students from enrolling in online courses even when they are not at elevated risk online. This suggests that institutions should be extremely cautious about implementing online readiness surveys before they have been rigorously tested for validity in predicting differential online versus face-to-face outcomes.