Quality of security metrics and measurements

•The results for prioritization of security metrics quality criteria are presented.•A conceptual security metrics quality criteria model is proposed.•The results derive from a security metrics expert study and an interview study.•Correctness, measurability, and meaningfulness are the foundational criteria.•In addition to the foundational criteria, usability should be emphasized.

Quantification of information security can be used to obtain evidence to support decision-making about the security performance of software systems. Knowledge about the relational importance of the main quality criteria of security metrics can help build security metrology models based on practical needs. This paper presents the results of a quantitative security metrics expert survey of 141 respondents, and an associated interview study, regarding the prioritization of 19 quality criteria of security metrics identified in the literature. The interviews were used to validate the survey results and to obtain further information on the findings. The results identified three foundational quality criteria of security metrics: correctness, measurability, and meaningfulness. These criteria form the basis for credibility and sufficiency for security metrics and associated measurements. Moreover, usability was seen as an important criterion. The paper analyzes the foundational and related quality criteria and proposes a model of them.