Examples

1. Minimizing runtime of a workload

It is possible to directly optimize the duration of a workload by issuing the following:

$ optimizer-studio ./workload.sh 
Concertio Optimizer Studio version 1.3
Optimizing ./workload.sh 
Optimization target: duration
Starting runtime (no time limit) ...

Optimization time: 01:26:27
Progress: 100%
Best known knob settings so far:
cpu.cache_prefetch: c
sys.vm.overcommit_ratio: 60
dev.raid.speed_limit_max: 250000
kernel.sched_wakeup_granularity_ns: 10000000

In this example, Optimizer Studio will measure the time it takes for ./workload.sh to complete and will attempt to minimize it by changing knob values and re-running. When no knobs.yaml file is specified, the default behavior of Optimizer Studio is to use its embedded knobs.

2. Maximizing reported benchmark performance

It is common to have a workload script run a benchmark and then report its performance. For example, running Phoronix's pts/apache benchmark reports the average number of web requests served per second. It is possible to directly maximize reported benchmark performance using Optimizer Studio. First, the benchmark performance needs to be extracted from ./workload.sh, for example:

$ cat workload.sh
#!/bin/bash
printf "n\n" | \
  phoronix-test-suite run pts/apache | \
  awk '/Average/{ print $2 }' | \
  sed "s,\x1B\[[0-9;]*[a-zA-Z],,g" > /tmp/result

The above will output the number of web requests into /tmp/result. We'll use this output in a custom knobs.yaml:

domain:
  common:
    metrics:
      my_target:
        kind: file
        path: /tmp/result
        aggregated: false
    include_metrics: [.*]
    plugins: [libfilemetricplugin.so]
    target: my_target:max

Optimizer Studio could then be launched as follows:

$ optimizer-studio --knobs=embedded --knobs=knobs.yaml ./workload.sh 

The order of the knob definition files is significant

Optimizer Studio loads knob definition files in the order they appear in the command line. However this doesn't include embedded knobs which are always processed first. This way your knob files have an opportunity to overload any embedded knobs settings.

3. Maximizing performance using asynchronous sampling

Optimizer Studio can improve the performance of applications by maximizing intermediate performance metrics, such as the current rate of retired instructions, the current rate of database queries, or other metrics that describe performance. In this scenario, Optimizer Studio will sample the metrics asynchronously and in parallel to the workload script. This can be accomplished by using an additional knobs.yaml file that contains a different target:

domain:
  common:
    target: performance:max

The above example knobs.yaml file will maximize the number of retired instructions per second performed by the CPU. Then, Optimizer Studio can be called as follows:

$ optimizer-studio --knobs=embedded --knobs=knobs.yaml --sampling-mode=async ./workload.sh 

In this example, the assumption is that the workload is simple and has only one phase of execution.

4. Optimizing a remote system

Optimizer Studio can optimize remote systems by using ssh. This is useful for optimizing BIOS settings on servers, optimizing non-x86 servers, or even optimizing network switches. This can be accomplished as follows:

  • In the workload script, run the workload on the remote device via ssh. Store the result on the local server running Optimizer Studio. The following example shows a workload script named workload.sh that calls a remote script named remote_workload.sh that resides on the remote device:
#!/bin/bash
ssh -i /home/ubuntu/.ssh/id_rsa ubuntu@52.91.47.2 \
    "./remote_workload.sh" > /tmp/result
  • Add a knobs.yaml file in which all knobs' set and get scripts use ssh to set/get knob values on the remote host. For example:
domain:
  common:
    knobs:
      my_remote_knob:
        options: [1, 2, 3]
        get_script: |
          ssh -i /home/ubuntu/.ssh/id_rsa \
              ubuntu@52.91.47.2 "cat /tmp/knob"
        set_script: |
          ssh -i /home/ubuntu/.ssh/id_rsa \
              ubuntu@52.91.47.2 "echo $KNOB_VAL > /tmp/knob"
    metrics:
      my_target:
        kind: file
        path: /tmp/result
        aggregated: false
    include_metrics: [.*]
    plugins: [libfilemetricplugin.so]
    target: my_target:max
  • Don't use the embedded knobs of Optimizer Studio as they will run on the local server and not on the remote device:
sudo optimizer-studio --knobs=knobs.yaml ./workload.sh 

5. Skewed distributions

Sometimes the distribution of the workload is not symmetric, or the optimization target is not to improve the average of the samples, but rather improve some predefined percentile (such as percentile 5 of the frames per second in a video game).

For such cases optimizer-studio offers, beyond the default average, two additional point estimators: * percentile - for selecting of a specific percentile. * mode - for estimating the value with the highest probability of occurring.

It should be noted that these point estimators cannot replace lack of samples. It is always good practice to configure the system to perform as much samples per configuration as possible without rendering the optimization work as too long.

For example, if our workload calculates the FPS of a given video game, we may wish to optimize the 5th percentile out of the sampled values. In such case, the configuration of the knobs.yaml file should include the following lines:

global_settings:
   point_estimator:
      percentile: 5

If you wish to select the value with the highest probability of occurrence, you can select the 'mode' estimator:

global_settings:
   point_estimator:
      mode:

It should also be noted that it is possible to set the point estimator to 'average:', but there is no need to do so as this is the default behavior.

6. Other use cases

Under /opt/concertio-optimizer/studio/examples/ there are additional examples:

$ ls -1
android_adb
a_plus_b
embedded_gcc
embedded_go
embedded_hadoop
embedded_hhvm
embedded_jvm
embedded_LAMP
embedded_mellanox_nic
embedded_mongodb
embedded_mysql
embedded_nginx
embedded_postgresql
embedded_redis
embedded_spark
evolution_synthetic
gromacs_mpi
hackbench_local
hackbench_remote
hadoop_runtime
jmeter_web
knob_dependency
noisy_async
parallel_ev_syntetic
parallel_ev_workload
parallel_ev_workload_ssh
phoronix
phoronix_parallel
reboot_workload
spec_gcc
spec_system
sum_a_b_c
sum_a_b_python
sum_knob_array
sum_knob_assoc_array
workload_in_yaml