Examples

Optimizer Studio package is installed with plenty of example experiments for user to pick from as a learning materiel and a jumpstart.

Usually it is a good practice to copy an example folder into user folder and start customizing it from there.

Following is a list of examples retrieved from version 3.14.0 of Optimizer-Studio. Under /opt/concertio-optimizer/studio/examples/:

$ ls -1
accelerators
android_adb
a_plus_b
a_plus_b_inventory
a_plus_b_wsl
blazemeter_nginx
compilers
datadog_a_plus_b
embedded_go
embedded_hadoop
embedded_hhvm
embedded_jemalloc
embedded_jvm
embedded_LAMP
embedded_mellanox_nic
embedded_mongodb
embedded_mysql
embedded_mysql_sysbench
embedded_nginx
embedded_postgresql
embedded_python
embedded_redis
embedded_spark
evolution_synthetic
gromacs_mpi
hackbench_local
hackbench_remote
hadoop_runtime
inventory_storage
jmeter_web
knob_dependency
newrelic_a_plus_b
noisy_async
parallel_ev_workload
parallel_ev_workload_ssh
phoronix
phoronix_parallel
reboot_workload
spec_aocc
spec_aocc_backend
spec_gcc
spec_system
sum_a_b_c
sum_a_b_c_async
sum_a_b_python
sum_knob_array
sum_knob_assoc_array
workload_in_yaml

Let's start with very simple use case, and then dive into several more sophisticated concepts from the above list:

1. Minimizing runtime of a workload while tuning OS settings

First, follow the steps to connect optimizer-studio with Conductor in the tutorial here.

It is possible to directly optimize the duration of a workload by defining the following knobs.yaml definition file and then running optimizer:

import:
  optimizer.studio:    # this import line tells optimizer-studio to use all the embedded applicable OS knobs on this device
workload:
  run:
    command: ./workload.sh    # this section defines what workload to run. In this case, a workload.sh script in local folder
domain:
  common:
    target: duration    # this line tells optimizer-studio to measure duration as the target metric for improvement

Since this use case tries to tune OS knobs, it is require to save knobs.yaml with root user as owner, and set the permissions as follows (assuming the file is in current folder):

sudo chown root:root ./knobs.yaml
sudo chmod g-w,o-w ./knobs.yaml

Assuming workload.sh workload script is also in current folder, simple run the following to start the experiment:

$ sudo optimizer-studio
INFO[0000] Defaulting to knobs file: /home/tomer/dev/tests/tst/knobs.yaml 
Concertio Optimizer Studio
Version 3.14.0
Running on Ubuntu 20.04
Testing the knob(s) ...
 Test all knobs. Summary:
 95 out of 95 knobs tested successfully, 0 knobs failed, 0 knobs not testable, 0 knobs to skip in tuning, 8 knobs disabled, 0 knobs failed to initialize
 Disabled knobs:
     app.numa.node.affinity
     symmetric_multithreading
     sys.block.nr_requests_nvme0n1
     sys.block.write_cache_nvme0n1
     sys.devices.system.clocksource
     sys.devices.system.cpu.smt.control
     thread.topology
     vm.zone_reclaim_mode
 All knob test: success
Knob testing completed successfully
 set stage: done
Starting service ...
...
Optimizing ./workload.sh 
Optimization target: duration
Starting runtime (no time limit) ...

Optimization time: 01:26:27
Progress: 100%
Best known knob settings so far:
cpu.cache_prefetch: c
sys.vm.overcommit_ratio: 60
dev.raid.speed_limit_max: 250000
kernel.sched_wakeup_granularity_ns: 10000000

In this example, Optimizer Studio will measure the time it takes for ./workload.sh to complete and will attempt to minimize it by changing knob values and re-running.

2. Maximizing reported benchmark performance

It is common to have a workload script run a benchmark and then report its performance. For example, running Phoronix's pts/apache benchmark reports the average number of web requests served per second. It is possible to directly maximize reported benchmark performance using Optimizer Studio.

First, the benchmark performance needs to be extracted from ./workload.sh, for example:

$ cat workload.sh
#!/bin/bash
printf "n\n" | \
  phoronix-test-suite run pts/apache | \
  awk '/Average/{ print $2 }' | \
  sed "s,\x1B\[[0-9;]*[a-zA-Z],,g" > /tmp/result

The above will output the number of web requests per second into /tmp/rps. We'll use this output in a custom knobs.yaml:

import:
  optimizer.studio:
domain:
  common:
    metrics:
      my_target:
        kind: file
        path: /tmp/rps
        aggregated: false
    include_metrics: [.*]
    plugins: [libfile-metrics.so]
    target: my_target:max
workload:
  kind: sync
  run:
    command: ./workload.sh
    timeout: {{best.duration * 2}}    # optional: stop experiment if the workload runs up to twice the time of the best result duration

Optimizer Studio could then be launched as follows:

$ optimizer-studio --knobs ./knobs.yaml         # note --knobs <file> is optional in case the file name is `knobs.yaml` and is in user environment PATH

3. Optimizing a remote system

Optimizer Studio can optimize remote systems by using ssh. This is useful for optimizing BIOS settings on servers, optimizing non-x86 servers, or even optimizing network switches. This can be accomplished as follows:

• In the workload script, run the workload on the remote device via ssh. Store the result on the local server running Optimizer Studio. The following example shows a workload script named workload.sh that calls a remote script named remote_workload.sh that resides on the remote device:

#!/bin/bash
ssh -i /home/ubuntu/.ssh/id_rsa ubuntu@52.91.47.2 \
    "./remote_workload.sh" > /tmp/result

• Add a knobs.yaml file in which all knobs' set and get scripts use ssh to set/get knob values on the remote host. For example:

domain:
  common:
    knobs:
      my_remote_knob:
        options: [1, 2, 3]
        get_script: |
          ssh -i /home/ubuntu/.ssh/id_rsa \
              ubuntu@52.91.47.2 "cat /tmp/knob"
        set_script: |
          ssh -i /home/ubuntu/.ssh/id_rsa \
              ubuntu@52.91.47.2 "echo $KNOB_VAL > /tmp/knob"
    metrics:
      my_target:
        kind: file
        path: /tmp/result
        aggregated: false
    include_metrics: [.*]
    plugins: [libfile-metrics.so]
    target: my_target:max
workload:
  run:
    command: ./workload.sh

• Don't use the embedded knobs of Optimizer Studio as they will run on the local server and not on the remote device:

sudo optimizer-studio --knobs ./knobs.yaml

4. Skewed distributions

Sometimes the distribution of the workload is not symmetric, or the optimization target is not to improve the average of the samples, but rather improve some predefined percentile (such as percentile 5 of the frames per second in a video game).

For such cases optimizer-studio offers, beyond the default average, two additional point estimators: * percentile - for selecting of a specific percentile. * mode - for estimating the value with the highest probability of occurring.

It should be noted that these point estimators cannot replace lack of samples. It is always good practice to configure the system to perform as much samples per configuration as possible without rendering the optimization work as too long.

For example, if our workload calculates the FPS of a given video game, we may wish to optimize the 5th percentile out of the sampled values. In such case, the configuration of the knobs.yaml file should include the following lines:

global_settings:
   point_estimator:
      percentile: 5

If you wish to select the value with the highest probability of occurrence, you can select the 'mode' estimator:

global_settings:
   point_estimator:
      mode:

It should also be noted that it is possible to set the point estimator to 'average:', but there is no need to do so as this is the default behavior.

5. Invalid knob detection and management

Optimizer Studio distribution contains an embedded library of knob files which can be used for different optimization scenarios. For example, optimization of GCC compiler flags for building a user application.

There exist some compiler flags which are valid for some versions and invalid for others. While the library supports different versions of compilers, it's impossible to support all the possible versions. In such cases, Optimizer Studio can recognize some flags (knobs) as invalid, and automatically exclude such knobs from subsequent knob configurations.

In order to switch on the active search for invalid knob combinations, include the following in your YAML configuration file:

domain:
  common:
    config_validation: </path/to/validity_file.yaml>

In case the file does not exist, Optimizer Studio will create it when running for the first time. Each subsequent run with the same YAML file will use and update the same validation file.