Cost, benefits and quality of software development documentation: A systematic mapping

• 38% of papers propose new techniques and 29% contribute empirical evidences.
• Validation research papers and solution proposals are dominating types.
• Agile development is only mentioned in six studies.
• Design, code and requirement are dominating document types.
• Formatted text, models and tool support are dominating document forms.
• Most papers had only one SUS whose sizes do not increase strictly over time.
• The average number of participants in survey-based studies was 106.
• 17% of papers focus on cost, in contrast to benefit (54%) and quality (72%).
• Only six studies (8%) discuss the development or production cost.
• Documentation's main usage includes maintenance aid and program comprehension.
• Most discussed quality attributes are completeness, consistency and accessibility.
• Only 10% of papers are affiliated with industry.

ContextSoftware documentation is an integral part of any software development process. Researchers and practitioners have expressed concerns about costs, benefits and quality of software documentation in practice. On the one hand, there is a lack of a comprehensive model to evaluate the quality of documentation. On the other hand, researchers and practitioners need to assess whether documentation cost outweighs its benefit.ObjectivesIn this study, we aim to summarize the existing literature and provide an overview of the field of software documentation cost, benefit and quality.MethodWe use the systematic-mapping methodology to map the existing body of knowledge related to software documentation cost, benefit and quality. To achieve our objectives, 11 Research Questions (RQ) are raised. The primary papers are carefully selected. After applying the inclusion and exclusion criteria, our study pool included a set of 69 papers from 1971 to 2011. A systematic map is developed and refined iteratively.ResultsWe present the results of a systematic mapping covering different research aspects related to software documentation cost, benefit and quality (RQ 1–11). Key findings include: (1) validation research papers are dominating (27 papers), followed by solution proposals (21 papers). (2) Most papers (61 out of 69) do not mention the development life-cycle model explicitly. Agile development is only mentioned in 6 papers. (3) Most papers include only one “System under Study” (SUS) which is mostly academic prototype. The average number of participants in survey-based papers is 106, the highest one having approximately 1000 participants. (4) In terms of focus of papers, 50 papers focused on documentation quality, followed by 37 papers on benefit, and 12 papers on documentation cost. (5) The quality attributes of documentation that appear in most papers are, in order: completeness, consistency and accessibility. Additionally, improved meta-models for documentation cost, benefit and quality are also presented. Furthermore, we have created an online paper repository of the primary papers analyzed and mapped during this study.ConclusionOur study results show that this research area is emerging but far from mature. Firstly, documentation cost aspect seems to have been neglected in the existing literature and there are no systematic methods or models to measure cost. Also, despite a substantial number of solutions proposed during the last 40 years, more and stronger empirical evidences are still needed to enhance our understanding of this area. In particular, what we expect includes (1) more validation or evaluation studies; (2) studies involving large-scale development projects, or from large number of study participants of various organizations; (3) more industry-academia collaborations; (4) more estimation models or methods to assess documentation quality, benefit and, especially, cost.