Finding profitable forecast combinations using probability scoring rules

This study examines the success of bets on Australian Football League (AFL) matches made by identifying panels of highly proficient forecasters and betting on the basis of their pooled opinions. The data set is unusual, in that all forecasts are in the form of probabilities. Bets are made “on paper” against quoted market betting odds according to the (fractional) Kelly criterion. To identify expertise, individual forecasters are scored using conventional probability scoring rules, a “Kelly score” representing the forecaster’s historical paper profits from Kelly-betting, and the more simplistic “categorical score” (number of misclassifications). Despite implicitly truncating all probabilities to either 0 or 1 before evaluation, and thus losing a lot of information, the categorical scoring rule appears to be a propitious way of ranking probability forecasters. Bootstrap significance tests indicate that this improvement is not attributable to chance.