P-boxes for cost uncertainty analysis

After years of developing point estimates analysts knew were uncertain, the estimating and acquisition community has embraced the concept of viewing a cost estimate as a potential distribution of cost represented by a Cumulative Distribution Function (CDF), commonly called the S-curve. Unfortunately, the very thing S-curves were intended to counter, i.e. the implication of preciseness that was inherent in point estimates for the cost, has become a preciseness about percentile values. Decision makers use the S-curve to make funding decisions. Decision makers, for example, have come to view the 50th percentile as an absolute and wonder why the number budgeted at 50% is shown as 30% when the estimate is updated. While there are many reasons for changes in the numbers, part of the error is inherent in the way the S-curve is developed. Ideally all input distributions in a cost estimate would be derived from reliable data and would have known shape and parameter values. In reality many inputs to an estimate are based on expert opinion and data of unknown relevance, making the distributions for these inputs uncertain. We can treat the S-curve, which is the CDF output of a cost analysis model, as an estimate of a “true” CDF, analogous to an Empirical Distribution Function (EDF), to help quantify the epistemic uncertainty inherent in the cost analysis process. Utilizing Kolmogorov–Smirnov and non-parametric quantile bounds, a p-box? is developed from which an analyst can now define a range of costs associated with specified levels of probability, or ranges in probability associated with specific costs.

► Epistemic uncertainty in DOD cost risk analysis is captured using p-boxes. ► Cost risk estimates (S-curves) are treated as empirical distribution functions (EDF). ► Common tools for bounding EDF's are used to create a p-box around the S-curve. ► An acquisition program's age is used to create a proxy value for observations.