Robustness and sensitivity of weighting and aggregation in constructing composite indices

A composite index is a combination of various sources of information known as indicators, measured in or of a system in order to provide a summary of the system that is itself not directly measurable. For the index to be useful and meaningful, its construction requires careful consideration of several important aspects of the potentially disparate and multiple indicators that help convey its meaning. In general there are five key steps that should be considered when constructing a composite index, one of which is how the indicators should be weighted and aggregated to form the index. This step is critical in index construction, yet there seems to be no documented evidence about making objective weighting and aggregation choices in constructing indices so that they are robust. This sort of evidence would be particularly helpful for deciding whether any of the existing indices for assessing stream health would suffice for assessing the health of a given set of stream data or whether developing a new index is warranted. Thus we designed a simulation study to test the robustness and sensitivity of the weighting and aggregation choices in four existing stream health indices. The four indices mainly differed in their choices of standardization and weighting and aggregation techniques. The three main general conclusions about these existing approaches are the recommendation to use bootstrapping to approximate the distribution of the stream health index; the standardization technique employed should use all of the available indicators; and the use of reference (or pristine) sites as a standardization tool is not essential. Since the study is based on artificial data, the findings should be applicable and relevant to indices in other fields of study such as economics, social sciences, finance and medicine.

► We describe the five key steps required in constructing a composite index. ► Weighting and aggregating indicators is a critical step in index construction. ► We simulated data to explore this step for 4 extant stream health indices. ► The robustness and sensitivity of the weighting and aggregating step was tested. ► The study helped assess the advantages and disadvantages of each approach.