The exception handling riddle: An empirical study on the Android API

We examine the use of the Java exception types in the Android platform's Application Programming Interface (API) reference documentation and their impact on the stability of Android applications. We develop a method that automatically assesses an API's quality regarding the exceptions listed in the API's documentation. We statically analyze ten versions of the Android platform's API (14-23) and 3539 Android applications to determine inconsistencies between exceptions that analysis can find in the source code and exceptions that are documented. We cross-check the analysis of the Android platform's API and applications with crash data from 901,274 application execution failures (crashes). We discover that almost 10% of the undocumented exceptions that static analysis can find in the Android platform's API source code manifest themselves in crashes. Additionally, we observe that 38% of the undocumented exceptions that developers use in their client applications to handle API methods also manifest themselves in crashes. These findings argue for documenting known might-thrown exceptions that lead to execution failures. However, a randomized controlled trial we run shows that relevant documentation improvements are ineffective and that making such exceptions checked is a more effective way for improving applications' stability.