A critical examination of recent industrial surveys on agile method usage

• Recent industrial surveys on agile method usage are presented and discussed.
• Quality of survey research is critically examined.
• Obtained results show insufficient thoroughness in reporting and trustworthiness.
• Recommendations are provided on how quality of survey research could be improved in the future.

ContextPractitioners and researchers often claim that agile methods have moved into the mainstream for the last few years. To support this claim they refer to recent industrial surveys which tend to report high rates of agile method usage. However many of these industrial surveys are conducted by agile consultants, tool vendors, professional societies and independent technology and market research organizations. This raises some important concerns about the possible conflict of interest and the overall trustworthiness of these studies.ObjectiveIn response to the above concerns, a secondary study was carried out. Its objective was to examine industrial surveys published in 2011 and 2012, determine the extent to which we could trust their reported high rates of agile method usage and provide recommendations on how quality of research could be improved in the future.MethodFollowing a rigorous search procedure, nine industrial surveys on agile method usage published in 2011 and 2012 were extracted from both academia and industry. Their thoroughness in reporting and trustworthiness were evaluated using a newly proposed assessment framework based on Guba's four attributes of trustworthiness (truth value, applicability, consistency and neutrality) and a number of methods for assessing survey research in related fields as information, communication and management studies.ResultsThe careful examination of the reviewed surveys shows that most of the studies have insufficient thoroughness in reporting and (subsequently) low trustworthiness. Only one (out of nine) study is considered as a scientific contribution in determining the current 2011/2012 rate of agile method usage.ConclusionsThe obtained results support our initial considerations about the trustworthiness of recent industrial surveys on agile method usage and suggest a number of recommendations for increasing the quality and value of future survey research in this regard.