Overview

The configuration files of Optimizer Runtime are located in the installation directory /opt/concertio-optimizer/runtime/. The default location can be overridden via command line switches.
There are two configuration files:

  1. knobs.yaml - Specifies the knobs, input metrics and target function of Optimizer Runtime.
  2. settings.yaml - Specifies the system-wide settings of Optimizer Runtime.

knobs.yaml: knob definition

Concertio Optimizer Runtime ships with an embedded knob file containing many knobs that were tested and benchmarked by Concertio engineers.
It is possible to provide additional knob files by using the --knobs=/path/to/knob-file.yaml command line switch. In the knob files one can add new custom knobs, and extend or override existing knobs.
While Optimizer Runtime can accept any number of --knobs=PATH command line switches, it attempts loading one by default: /opt/concertio-optimizer/runtime/knobs.yaml, this file contains a convenient knob file template.
Users are encouraged to add custom knobs that are relevant for their systems.
Note, that complete disabling of the embedded knob file is triggered by --knobs=no-embedded command line switch.

knobs section: adding a Knob

Adding or overriding an existing knob is possible by adding a knob section to the knobs list. Each knob section contains a few values and POSIX shell scripts used for defining and manipulating the knob. Each knob script is allowed to use the following environment variables:

  1. $KNOB_VAL - knob value being set by set_script (below)
  2. $KNOB_DEV - device name, in device-specific knobs, as returned by devices_script (below)

Mandatory knob scripts

  1. get_script - returns current knob value
  2. set_script - sets new knob value

Optional knob scripts

  1. pre_set_script - when provided, this script is run prior to running any of the set_script(s)
    Note that in case more than one knob refers to the same pre_set_script, identical script invocations will be merged.
  2. post_set_script - when provided, this script is run after running any of the set_script(s)
    Note that in case more than one knob refers to the same post_set_script, identical script invocations will be merged.
  3. enable_script - when provided, returns 0 to indicate that the knob should be discarded, or any other value to enable the knob. When not provided, the knob is enabled by default.
  4. device_script - when provided, returns list of applicable devices, and the knob is split into multiple per-device knobs, e.g.
    io.scheduler → io.scheduler.sda, io.scheduler.sdb, io.scheduler.sdc

Optional knob parameters

  1. options - a list of available knob options
  2. disable - when provided, the knob will be discarded and erased from internal data structures. This parameter overrides enable_script (above).
  3. skip_tuning - when provided, the knob will be excluded from the tuning process
  4. default - when provided, sets the baseline default to the value provided. this option is not allowed for knobs with get script

Knob options definition

The simplest way to define knob options is to list all valid values explicitly:

options:
  values: [10, 15, 20]
default: 15

The second way is to define an options script which returns the list of option values. For example, the following script creates a list of numbers from 0 to 7 for a computer with 8 logical CPUs

options:
  script: cat /proc/cpuinfo | awk '/processor/ {print $3}'

Yet another possibility is to define a list of options using numeric range:

options:
  range:
    min: 1
    max: 16
    step: 1
    format: --num-threads=%d
default: 8

This definition creates a list of options "--num-threads=1", --num-threads=2, etc.
Printf style format field is optional. It contains a string with one of supported format specifiers, like in the example:

Format Numeric Value Knob Value
"-O%d" 2 -O2
"--pi=%.2f" 3.14159 --pi=3.14
"--pi=%.2f" 3 --pi=3.00
"%g" 5000000 5000000
"%g" 2.718000 2.718
"%x" 2047 7ff
"%X" 2047 7FF

You can combine different ways of specification:

options:
  values: [ "" ]
  range:
    min: 1
    max: 16
    step: 1
    format: --num-threads=%d

The example above adds an empty option value in addition to the ones defined by the range.

Disabling a knob

It is possible to disable a knob, either temporarily or permanently, without actually removing its definition from the knob file.
To temporarily disable a knob, add to the knob section the following value:

skip_tuning: ""

To permanently erase a knob from the internal data structures, either add to the knob section the following value:

disable: ""

or define an appropriate enable_script:

enable_script: "echo -n 0"

Note that this approach allows disabling embedded knobs as well.

scripts section: named reusable scripts

Frequently different knobs reuse the same scripts, in particular pre_set_script(s)/post_set_script(s), as these are intended to preface/conclude multiple knobs.
In the scripts section, reusable scripts are represented as name:code pairs. Instead of providing the full code, it is possible to refer the script name prefixed with a @ symbol, e.g. "@my-script".

Examples

A: vm.dirty_background_ratio

domain:
  common:
    knobs:
      sys.vm.dirty_background_ratio:
        description: "The number of pages at which the pdflush threads begin writeback of dirty data."
        get_script: "/sbin/sysctl -n vm.dirty_background_ratio"
        set_script: "/sbin/sysctl -w vm.dirty_background_ratio=$KNOB_VAL"
        options: [10, 15, 20]

B: io.scheduler

Below is a more complex knob example of a device-specific knob. Eventually, this knob definition produces multiple knobs: io.scheduler.sda, io.scheduler.sdb etc.

domain:
  common:
    knobs:
      io.scheduler:
        description: "Block device scheduling algorithm selector."
        get_script: "cat /sys/block/$KNOB_DEV/queue/scheduler | sed 's/.*\\[\\([^]]*\\)\\].*/\\1/g'"
        set_script: "echo $KNOB_VAL > /sys/block/$KNOB_DEV/queue/scheduler"
        device_script: "ls -d /sys/block/sd* | cut -d/ -f 4"
        options:
          script: cat /sys/block/$KNOB_DEV/queue/scheduler | sed 's/\[\|\]//g ; s/ $//g ; s/\s/\n/g'

metrics section: named HW and SW metrics

Metrics are used by Optimizer Runtime to learn about the system behavior and to detect different phases of execution. Optimizer Runtime will then attempt to find an optimal knob configuration that maximizes a certain sampled metric for each phase. The metrics are sampled periodically.

Metrics definition

Comma separated regular expressions define which metrics are sampled, and which metrics are excluded.

domain:
  common:
    include_metrics: [msr.*, proc.*]
    exclude_metrics: [proc.diskstats.sda.sectors_written]

In the above example, all msr metrics and all proc metrics, except for proc.diskstats.sda.sectors_written will be considered by Optimizer Runtime for learning about the system behavior.

User-defined metrics

Optimizer Runtime supports user-developed plugins for sampling custom metrics.

Target metric

Users can configure Optimizer Runtime to search for the optimal parameters that maximize a specific target metric.

domain:
  common:
    target: performance

The target metric can be one of the following:

  1. performance - defined as the total number of retired instructions msr.inst_retired_all measured in all CPUs per second.
  2. energy - defined as the package energy of the CPUs per retired instruction. In case of energy, Optimizer Runtime will attempt to minimize the energy per retired instruction.
  3. Any other available metric, as long as it is sampled by Optimizer Runtime, such as proc.diskstats.sda.sectors_written.

settings.yaml: system-wide settings

This file contains runtime settings such as optimization target function, sampling rate, output directory, etc.

Output directory

Concertio Optimizer Runtime generates output data such as optimization database file, log files, etc. The default location of all output data is /root/.concertio. The default location can be overridden as follows:

out_directory: "/new/output/directory"

Available Setting Parameters

Parameter name Default value Description
sampling_interval 1s The interval between samples. Relevant only for asynchronous sampling mode.
knob_ranking_max_num_of_samples 10000 Maximum number of configurations to use for knob ranking calculations.
max_baseline_cv 0.04 The maximum allowed coefficient of variation of the mean of the measurements in baseline settings. Lower values imply a stricter convergence threshold, so additional measurements might be required in order to converge.
max_config_mean_cv 0.04 The maximum allowed coefficient of variation of the mean of the measurements per knob configuration. Lower values imply a stricter convergence threshold, so additional measurements might be required in order to converge.
max_configs_in_report 10 Maximum number of knob configurations to include into best configurations report
max_invalid_samples_per_config 0 The maximum allowed invalid measurements per knob configuration, above which the configuration is considered invalid and will not be further tested.
max_samples_per_config 120 Optimizer will not test any knob configuration more than the number of times specified by this parameter.
metrics_csv_filename - If specified, Optimizer Studio creates a CSV file with the details of all the knob settings and metric measurements.
min_baseline_samples 2 The minimum number of baseline samples. This is used in conjunction with max_baseline_cv.
min_samples_per_config 2 The minimum number of samples per knob configuration. This is used in conjunction with max_config_mean_cv.
optimization_strategy evolution The algorithm employed for searching through the knobs. Available options are greedy and evolution.
optimization_strategy_settings Settings specific for each optimization strategy. Only settings specific for the selected strategy will be parsed.
out_directory ${HOME}/.concertio Concertio Optimizer Studio generates output data such as optimization database file, log files, etc. into this location.
pending_config_timeout 0 Configuration attempt scheduling policy: 0 - attempt configs sequentially, above 0 (in minutes) - attempt configs interleaved with a timeout.
point_estimator average: <no_value> Point estimation function. Additional functions: percentile: <percent>, mode: <no_value>
save_interval 120m The interval between saving the data file to the disk
shell_command /bin/sh +e This defines the backend shell of the knobs and metrics. It is possible to run all knobs and metrics scripts on remote hosts using a different shell command
max_idle_time 10m maximum time for optimizer service to live without any connection from clients after this time the service exits. default value if not set in settings.yaml is 0 meaning no timeout

Deprecated Setting Parameters

If you get a parameter deprecation warning when running Optimizer Studio, please replace the deprecated parameters with new ones in your YAML files, adjusting the parameter values accordingly.

Note that the old parameters still work as before. However they will be dropped in the future versions.

Old parameter New parameter Change Example
interval_seconds sampling_interval Time units 1s
save_interval_minutes save_interval Time units 2h
pending_config_timeout_minutes pending_config_timeout Time units 30m
max_idle_time_minutes max_idle_time Time units 10m