Skip to content

Overview

The configuration files of Optimizer Runtime are located in the installation directory /opt/concertio-optimizer/runtime/. The default location can be overridden via command line switches.
There are two configuration files:

  1. knobs.yaml - Specifies the knobs, input metrics and target function of Optimizer Runtime.
  2. settings.yaml - Specifies the system-wide settings of Optimizer Runtime.

Optimization Target

Users can configure Optimizer Runtime to search for the optimal parameters that maximize (or minimize) a specific Target Metric.
The target metric can be defined as follows:

domain:
  common:
    target:
      name: performance
      goal: max

or in shorthand:

domain:
  common:
    target: performance:max

The target metric name can refer any of the sampled metrics, such as proc.diskstats.sda.sectors_written. The metrics definition is further detailed below.
The optimization goal is set to max, if omitted.

Predefined Optimization Target

While target metric can refer any of the sampled metrics, Optimizer-Runtime supports a list of predefined targets:

Name Default Goal Meaning
duration min Total workload execution time.
performance max Total number of retired instructions (msr.inst_retired_all) measured in all CPU cores per second.
net.throughput max Total number of bytes transferred over the network per second.

knobs.yaml: knob definition

Knobs are tunable parameters of the system which Optimizer Runtime will try to change in order to optimize the system.

Concertio Optimizer Runtime ships with embedded set of knob definitions containing many knobs that were tested and benchmarked by Concertio engineers. In order to use these knobs, specify --knobs=embedded command line switch.

It is possible to provide additional knob files by using the --knobs=/path/to/knob-file.yaml command line switch, where you can add custom knobs that are relevant for your system. If you don't supply your knobs file, the embedded knobs will be used automatically.

Optimizer Runtime can accept any number of --knobs=PATH command line switches. In such case Optimizer Runtime loads knob files in the order they appear in the command line. This way a knob files appearing later can overload any settings in previous knob files. Note that this doesn't include embedded knobs which are always processed first.

knobs section: adding a Knob

It is possible to add a new or override an existing knob by adding a knob section to the knobs list. Each knob section contains a few values and POSIX shell scripts used for defining and manipulating the knob.

Knob kinds

Each knob belongs to one of the supported knob kinds: builtin, scripted, accelerator.
The main difference between the knob kinds is the way the knob is applied.
Below is a list of the supported knob kinds:

  1. builtin knobs
    Knobs built into Optimizer Runtime do not need to be defined by the user, but might require configuration parameters.

  2. scripted knobs
    The scripted knob is applied via executing a POSIX shell script with $KNOB_VAL, $KNOB_DEV environment variables.

    • kind: script
    • get_script - returns the current knob value, that will be set as baseline
    • set_script - set new knob value
    • pre_set_script - when provided, this script is run prior to running any of the set_script(s)
      Note that in case more than one knob refers to the same pre_set_script, identical script invocations will be merged.
    • post_set_script - when provided, this script is run after running any of the set_script(s)
      Note that in case more than one knob refers to the same post_set_script, identical script invocations will be merged.
    • device_script - when provided, returns the list of applicable devices, and the knob is split into multiple per-device knobs, e.g.
      io.scheduler → io.scheduler.sda, io.scheduler.sdb, io.scheduler.sdc
    • Environment variables observable by the knob scripts:
      $KNOB_VAL - knob value being set by set_script
      $KNOB_DEV - device name, in device-specific knobs, as returned by device_script
  3. accelerator knobs
    The accelerator knobs encompass all the knob kinds passed directly to Concertio accelerator.

    • kind: accel.mem_alloc - provides a selection of malloc implementations

General knob parameters

Below is a list of the general purpose parameters supported by all the knob kinds:

  1. kind - knob kind. When omitted, Optimizer software will attempt to identify the knob kind by presence or absence of kind-specific parameters.
  2. enable_script - when provided, a 0 / false indicate that the knob should be discarded, any other value enables the knob. When not provided, the knob is enabled by default.
  3. disable - when provided, the knob will be discarded and erased from internal data structures. This parameter overrides enable_script.
  4. options - a list of available knob options. Options definition is detailed below.
  5. default - when provided, sets the baseline to the value provided. This option is mutually exclusive with get_script.
    Notice, that when neither default nor get_script are specified, the 1-st option in the options list will be set as baseline value.
  6. set_policy - knob setting policy:

    • on_change (default) - each time the internal algorithm needs to change a knob, its value is applied.
    • once:<value> - the value is applied in the beginning of the run, and is reverted to baseline in the end.
    • always - the current knob value is applied on each sample.

    Once and always setting policies do not participate in the optimization process, and can be considered auxiliary utilities.

Knob options definition

The simplest way to define knob options is to list all valid values explicitly:

options:
  values: [10, 15, 20]
default: 15

The second way is to define an options script which returns the list of option values. For example, the following script creates a list of numbers from 0 to 7 for a computer with 8 logical CPUs

options:
  script: cat /proc/cpuinfo | awk '/processor/ {print $3}'

Yet another possibility is to define a list of options using numeric range:

options:
  range:
    min: 1
    max: 16
    step: 1
    format: --num-threads=%d
default: 8

This definition creates a list of options "--num-threads=1", --num-threads=2, etc.
Printf style format field is optional. It contains a string with one of supported format specifiers, like in the example:

Format Numeric Value Knob Value
"-O%d" 2 -O2
"--pi=%.2f" 3.14159 --pi=3.14
"--pi=%.2f" 3 --pi=3.00
"%g" 5000000 5000000
"%g" 2.718000 2.718
"%x" 2047 7ff
"%X" 2047 7FF

You can combine different ways of specification:

options:
  values: [ "" ]
  range:
    min: 1
    max: 16
    step: 1
    format: --num-threads=%d

The example above adds an empty option value in addition to the ones defined by the range.

Disabling a knob

It is possible to disable a knob, either temporarily or permanently, without actually removing its definition from the knob file.
To temporarily disable a knob, add to the knob section the following value:

skip_tuning: ""

To permanently erase a knob from the internal data structures, either add to the knob section the following value:

disable: ""

or define an appropriate enable_script:

enable_script: "echo -n 0"

Note that this approach allows disabling embedded knobs as well.

Examples

A: vm.dirty_background_ratio

domain:
  common:
    knobs:
      sys.vm.dirty_background_ratio:
        kind: script
        description: "The number of pages at which the pdflush threads begin writeback of dirty data."
        get_script: "/sbin/sysctl -n vm.dirty_background_ratio"
        set_script: "/sbin/sysctl -w vm.dirty_background_ratio=$KNOB_VAL"
        options: [10, 15, 20]

B: io.scheduler

Below is a more complex knob example of a device-specific knob. Eventually, this knob definition produces multiple knobs: io.scheduler.sda, io.scheduler.sdb etc.

domain:
  common:
    knobs:
      io.scheduler:
        kind: script
        description: "Block device scheduling algorithm selector."
        get_script: "cat /sys/block/$KNOB_DEV/queue/scheduler | sed 's/.*\\[\\([^]]*\\)\\].*/\\1/g'"
        set_script: "echo $KNOB_VAL > /sys/block/$KNOB_DEV/queue/scheduler"
        device_script: "ls -d /sys/block/sd* | cut -d/ -f 4"
        options:
          script: cat /sys/block/$KNOB_DEV/queue/scheduler | sed 's/\[\|\]//g ; s/ $//g ; s/\s/\n/g'

metrics section: named HW and SW metrics

Metrics are used by Optimizer Runtime to learn about the system behavior and to detect different phases of execution. Optimizer Runtime will then attempt to find an optimal knob configuration that maximizes a certain sampled metric for each phase. The metrics are sampled periodically.

Metrics definition

Comma separated regular expressions define which metrics are sampled, and which metrics are excluded.

domain:
  common:
    include_metrics: [msr.*, proc.*]
    exclude_metrics: [proc.diskstats.sda.sectors_written]

In the above example, all msr metrics and all proc metrics, except for proc.diskstats.sda.sectors_written will be considered by Optimizer Runtime for learning about the system behavior.

User-defined metrics

Optimizer Runtime supports user-developed plugins for sampling custom metrics.

Categorized user-defined metrics

In version 1.1.0 of optimizer-runtime, a new feature allows the possibility to define metrics which will serve as categorial for workload classification. To make it more comfortable, optimizer supports metrics which return a string rather than only numeric value. This allows for more human-readable category based metrics to be defined.

Following is syntax for declaring such metrics as categorizing metrics using an array for optimizer-runtime to classify category based workloads:

domain:
  common:
    metrics:
      load_avg:
      ...
    categorization_metrics:
    - load_avg    

settings.yaml: system-wide settings

This file contains runtime settings such as optimization target function, sampling rate, output directory, etc.

Output directory

Concertio Optimizer Runtime generates output data such as optimization database file, log files, etc. The default location of all output data is /root/.concertio. The default location can be overridden as follows:

out_directory: "/new/output/directory"

Importing In-Situ Optimizer-Studio Knobs

If you ran an in-situ experiment using Optimizer Studio and you wish to import the best settings of this experiment into your Optimizer Runtime environment, you can use the import_accel_knobs command of the optimizer-ctl tool.

This command receives a file path to a json report of the relevant experiment (should be available in your Optimizer Studio output directory) as well as a file path for your target YAML configuration file which will be created.

For example:

$ optimizer-ctl import_accel_knobs --report ~/.concertio/report_2021-02-01_11-09-28_1014287.json --dest insitu.yaml

This will create the configuration file insitu.yaml using the results in ~/.concertio/report_2021-02-01_11-09-28_1014287.json.

Depending on the configuration used in the Optimizer Studio experiment, the content of the output file (insitu.yaml) may look like:

domain:
  common:
    knobs:
      memory:
        kind: accel.mem_alloc
        options:
          - ""
          - mimalloc
        set_policy:
          once: mimalloc

Note: All the knobs in the output configuration file will use set_policy of once. This is because these knobs cannot be changened during the entire session of Optimizer Runtime.

You should use the generated configuration file as either a base for your Optimizer Runtime configuration or you may import this file from your already available configuration file.

Available Setting Parameters

Parameter name Default value Description
sampling_interval 1s The interval between samples. Relevant only for asynchronous sampling mode.
knob_ranking_max_num_of_samples 10000 Maximum number of configurations to use for knob ranking calculations.
max_baseline_cv 0.04 The maximum allowed coefficient of variation of the mean of the measurements in baseline settings. Lower values imply a stricter convergence threshold, so additional measurements might be required in order to converge.
max_config_mean_cv 0.04 The maximum allowed coefficient of variation of the mean of the measurements per knob configuration. Lower values imply a stricter convergence threshold, so additional measurements might be required in order to converge.
max_configs_in_report 10 Maximum number of knob configurations to include into best configurations report
max_invalid_samples_per_config 0 The maximum allowed invalid measurements per knob configuration, above which the configuration is considered invalid and will not be further tested.
max_samples_per_config 120 Optimizer will not test any knob configuration more than the number of times specified by this parameter.
metrics_csv_filename - If specified, Optimizer Runtime creates a CSV file with the details of all the knob settings and metric measurements.
min_baseline_samples 2 The minimum number of baseline samples. This is used in conjunction with max_baseline_cv.
min_samples_per_config 2 The minimum number of samples per knob configuration. This is used in conjunction with max_config_mean_cv.
optimization_strategy evolution The algorithm employed for searching through the knobs. Available options are greedy and evolution.
optimization_strategy_settings Settings specific for each optimization strategy. Only settings specific for the selected strategy will be parsed.
out_directory ${HOME}/.concertio Concertio Optimizer Runtime generates output data such as optimization database file, log files, etc. into this location.
pending_config_timeout 0 Configuration attempt scheduling policy: 0 - attempt configs sequentially, above 0 (in minutes) - attempt configs interleaved with a timeout.
point_estimator average: <no_value> Point estimation function. Additional functions: percentile: <percent>, mode: <no_value>
save_interval 120m The interval between saving the data file to the disk
shell_command /bin/sh +e This defines the backend shell of the knobs and metrics. It is possible to run all knobs and metrics scripts on remote hosts using a different shell command
max_idle_time 10m maximum time for optimizer service to live without any connection from clients after this time the service exits. default value if not set in settings.yaml is 0 meaning no timeout

Deprecated Setting Parameters

If you get a parameter deprecation warning when running Optimizer Runtime, please replace the deprecated parameters with new ones in your YAML files, adjusting the parameter values accordingly.

Note that the old parameters still work as before. However they will be dropped in the future versions.

Old parameter New parameter Change Example
interval_seconds sampling_interval Time units 1s
save_interval_minutes save_interval Time units 2h
pending_config_timeout_minutes pending_config_timeout Time units 30m
max_idle_time_minutes max_idle_time Time units 10m