Comparing the accuracy of default predictions in the rating industry for different sets of obligors

•We show how probability forecasts can be ranked for different sets of events.•This ranking generalizes the refinement ordering which is only applicable to identical sets of events.•This ranking provides a partial ordering which is consistent with popular skill scores used in practice.

We generalize the refinement ordering for well calibrated probability forecasts to the case were the debtors under consideration are not necessarily identical. This ordering is consistent with many well known skill scores used in practice. We also add an illustration using default predictions made by the leading rating agencies Moody's and S&P.