Summary

Cloud service benchmarking can provide important, sometimes surprising insights into the quality of services and leads to a more quality-driven design and engineering of complex software architectures that use such services. Starting with a broad introduction to the field, this book guides readers step-by-step through the process of designing, implementing and executing a cloud service benchmark, as well as understanding and dealing with its results. It covers all aspects of cloud service benchmarking, i.e., both benchmarking the cloud and benchmarking in the cloud, at a basic level.

The book is divided into five parts: Part I discusses what cloud benchmarking is, provides an overview of cloud services and their key properties, and describes the notion of a cloud system and cloud-service quality. It also addresses the benchmarking lifecycle and the motivations behind running benchmarks in particular phases of an application lifecycle. Part II then focuses on benchmark design by discussing key objectives (e.g., repeatability, fairness, or understandability) and defining metrics and measurement methods, and by giving advice on developing own measurement methods and metrics. Next, Part III explores benchmark execution and implementation challenges and objectives as well as aspects like runtime monitoring and result collection. Subsequently, Part IV addresses benchmark results, covering topics such as an abstract process for turning data into insights, data preprocessing, and basic data analysis methods. Lastly, Part V concludes the book with a summary, suggestions for further reading and pointers to benchmarking tools available on the Web.

The book is intended for researchers and graduate students of computer science and related subjects looking for an introduction to benchmarking cloud services, but also for industry practitioners who are interested in evaluating the quality of cloud services or who want to assess key qualities of their own implementations through cloud-based experiments.

Authors

• David Bermbach is a Senior Researcher in the Information Systems Engineering Research Group at TU Berlin, Germany. His main research interests are in cloud and fog computing with a focus on distributed data management. David has a PhD in computer science and a diploma in business engineering both from Karlsruhe Institute of Technology, Germany.
• Erik Wittern is a Research Staff Member at the IBM Thomas J. Watson Research Center in New York, USA. His research interests are in the areas of cloud computing and software engineering - specifically, he is concerned with supporting developers in providing and consuming web APIs. Prior to joining IBM Research, Erik completed his PhD in computer science at the Karlsruhe Institute of Technology, Germany.
• Stefan Tai is Professor and Head of Chair Information Systems Engineering at TU Berlin, Germany. His principal research interests are in the areas of cloud computing, distributed systems, and software service engineering. Prior to his appointment in Berlin, Stefan was a Professor at Karlsruhe Institute of Technology and a Research Staff Member at the IBM Thomas J. Watson Research Center in New York, USA.