Overview

In order to evaluate the benefits of using Optimizer Runtime, a benchmarking package is available.
The benchmarking package allows running Phoronix Test Suite benchmarks or user-defined benchmarks. The benchmarking script evaluates the performance of a benchmark in four different phases:

  1. Baseline - Optimizer Runtime is shut down.
  2. Learning - Optimizer Runtime is activated with --mode=tune command line switch. In this mode Optimizer Runtime studies the system workload and explores the parameter space of the system knobs searching for the best configuration.
  3. Optimized - Optimizer Runtime is activated with --mode=best command line switch. In this mode the best configurations found during the Learning phase are applied.
  4. Static - Optimizer Runtime is activated with --mode=static-best command line switch. In this mode the best configurations found during the Learning phase are applied and then Optimizer Runtime exits.

Upon completion of several iterations in each phase, the benchmark script prints a summary which includes:

  1. The average performance measured in each phase.
  2. The relative performance improvement in each phase over the baseline phase.
  3. The relative standard deviation measured in each phase.
  4. An SVG graph plot file of the results, saved in /tmp/plot.svg
  5. Optional self extractable executable of optimizer database, for replicating static-best tuning database to other machines.

Usage

The Optimizer Runtime benchmarking package includes the optimizer-benchmark script. Running the script with no arguments will display a brief help screen, and a list of currently supported benchmarks:

$ optimizer-benchmark
Usage: optimizer-benchmark [--prepare-static] <N>[/N/N/N] <Benchmark>

Concertio Optimizer Runtime benchmark suite.
Boosting server performance and energy efficiency.

Command line options:
  prepare-static    prepare the optimizer-package-db file and load it with what it learned
  N                 (the same) number of iterations in each test phase
  N/N/N/N           number of iterations per test phase: Baseline/Learning/Optimization/Static
  Benchmark         benchmark name

Supported benchmarks:
  pts/apache        Phoronix tests suite apache
  custom            Run a custom test specified in the command line, e.g.
                    $ optimizer-benchmark 1 custom sleep 1

Running a benchmark

In order to run one of the available benchmarks, issue

$ sudo optimizer-benchmark <N>[/N/N/N] <Benchmark>

Where

  • N is the number of times the benchmark will be repeated during each phase. The N/N/N/N format enables setting a different number of repetitions per phase.
  • Benchmark is either one of the pre-defined benchmark names or custom, which allows running a custom benchmark

Example usage

Below is an example usage of the Phoronix Apache benchmark pts/apache, executed 5 times in Baseline, 80 times in Learning, 10 times in Optimized and 5 times in Static operating modes:

$ sudo optimizer-benchmark 5/80/10/5 pts/apache
Set up benchmark suite ...
Start benchmark ...
Baseline Phase:      [########################################] 100%
Learning Phase:      [########################################] 100%
Optimization Phase:  [########################################] 100%
Static Phase:        [########################################] 100%
Summary for benchmark "pts/apache": 
phase        #runs  performance    improvement    stdev          
Baseline     5           5972.48            0%        1.314% 
Learning     80          6949.23       16.354%        1.949% 
Optimized    10           7027.4       17.662%        0.466% 
Static       5           7075.81       18.473%         0.18% 
plot saved at /tmp/plot.svg

Below is an example of running a user-defined benchmark, in this case a simple one-second sleep, executed twice in each phase:

$ sudo optimizer-benchmark --prepare-static 2 custom sleep 1
Start benchmark ...
Baseline Phase:      [########################################] 100%
Learning Phase:      [########################################] 100%
Optimization Phase:  [########################################] 100%
Static Phase:        [########################################] 100%
Summary for benchmark "sleep 1": 
phase        #runs  performance    improvement    stdev          
Baseline     2           1.00663            0%        0.444% 
Learning     2           1.00659        0.003%        0.627% 
Optimized    2            1.0065        0.012%            0% 
Static       2           1.00691       -0.027%            0% 
plot saved at /tmp/plot.svg
opt-data collected into package at: /tmp/optimizer-package-db. don't forget to remove it once copied to other machines

Output file

The raw output of running Phoronix benchmark is dumped to /tmp/optimizer-benchmark.log.

See Also