An empirical study on the quantitative notion of task difficulty

• Analysis of differences between tasks difficulty models in ITS and IRT.
• Comparison between statistical and heuristic estimation of questions difficulty.
• Teachers’ estimation of difficulty is not always reliable.
• Students’ estimation is more accurate than teacher’s estimation.
• Human difficulty estimation is closer to the IRT 3PL model.

Most Adaptive and Intelligent Web-based Educational Systems (AIWBES) use tasks in order to collect evidence for inferring knowledge states and adapt the learning process appropriately. To this end, it is important to determine the difficulty of tasks posed to the student. In most situations, difficulty values are directly provided by one or more persons. In this paper we explore the relationship between task difficulty estimations made by two different types of individuals, teachers and students, and compare these values with those estimated from experimental data. We have performed three different experiments with three different real student samples. All these experiments have been done using the SIETTE web-based assessment system. We conclude that heuristic estimation is not always the best solution and claim that automatic estimation should improve the performance of AIWBES.