An empirical study for evaluating the performance of multi-cloud APIs

The massive use of cloud APIs for workload orchestration and the increased adoption of multiple cloud platforms prompted the rise of multi-cloud APIs. Multi-cloud APIs abstract cloud differences and provide a single interface regardless of the target cloud platform. Identifying whether the performance of multi-cloud APIs differs significantly from platform-specific APIs is central for driving technological decisions on cloud applications that require maximum performance when using multiple clouds. This study aims to evaluate the performance of multi-cloud APIs when compared to platform-specific APIs. We carried out three rigorous quasi-experiments to measure the performance (dependent variable) of cloud APIs (independent variable) regarding CPU time, memory consumption and response time. jclouds and Libcloud were the two multi-cloud APIs used (experimental treatment). Their performance were compared to platform-specific APIs (control treatment) provided by Amazon Web Services and Microsoft Azure. These APIs were used for uploading and downloading (tasks) 39 722 files in five different sizes to/from storage services during five days (trials). Whereas jclouds performed significantly worse than platform-specific APIs for all performance indicators on both cloud platforms and operations for all five file sizes, Libcloud outperformed platform-specific APIs in most tests (p-value not exceeding 0.00125, A-statistic greater than 0.64). Once confirmed by independent replications, our results suggest that jclouds developers should review the API design to ensure minimal overhead whereas jclouds users should evaluate the extent to which this trade-off affect the performance of their applications. Multi-cloud users should carefully evaluate what quality attribute is more important when selecting a cloud API.