Using Microbenchmark Suites to Detect Application Performance Changes

by   Martin Grambow, et al.

Software performance changes are costly and often hard to detect pre-release. Similar to software testing frameworks, either application benchmarks or microbenchmarks can be integrated into quality assurance pipelines to detect performance changes before releasing a new application version. Unfortunately, extensive benchmarking studies usually take several hours which is problematic when examining dozens of daily code changes in detail; hence, trade-offs have to be made. Optimized microbenchmark suites, which only include a small subset of the full suite, are a potential solution for this problem, given that they still reliably detect the majority of the application performance changes such as an increased request latency. It is, however, unclear whether microbenchmarks and application benchmarks detect the same performance problems and one can be a proxy for the other. In this paper, we explore whether microbenchmark suites can detect the same application performance changes as an application benchmark. For this, we run extensive benchmark experiments with both the complete and the optimized microbenchmark suites of the two time-series database systems InuxDB and VictoriaMetrics and compare their results to the results of corresponding application benchmarks. We do this for 70 and 110 commits, respectively. Our results show that it is possible to detect application performance changes using an optimized microbenchmark suite if frequent false-positive alarms can be tolerated.


page 1

page 2

page 3

page 4


An Approach and Benchmark to Detect Behavioral Changes of Commits in Continuous Integration

When a developer pushes a change to an application's codebase, in the fo...

Creating a Virtuous Cycle in Performance Testing at MongoDB

It is important to detect changes in software performance during develop...

Change Point Detection in Software Performance Testing

We describe our process for automatic detection of performance changes f...

Multi-Objective Search-Based Software Microbenchmark Prioritization

Ensuring that software performance does not degrade after a code change ...

Automated Identification of Performance Changes at Code Level

To develop software with optimal performance, even small performance cha...

EigenEvent: An Algorithm for Event Detection from Complex Data Streams in Syndromic Surveillance

Syndromic surveillance systems continuously monitor multiple pre-diagnos...

Detection of small changes in medical and random-dot images comparing self-organizing map performance to human detection

Radiologists use time series of medical images to monitor the progressio...

Please sign up or login with your details

Forgot password? Click here to reset