Skip to content

Workload Definition

Workload defined via command line

The simplest way to define Optimizer Studio workload is by providing a workload script on the command line:

$ optimizer-studio --knobs /path/to/knobs.yaml /path/to/workload.sh

The responsibility of the workload script is to export its performance to the target metric, usually by storing its value in a file.
The target metric is sampled upon workload script completion.

Workload declarative definition

To better control integration of workload execution with Optimizer Studio, Concertio provides the workload declarative syntax in the experiment defintion file (knobs.yaml).
The workload definition contains a structured recipe of how Optimizer Studio should execute, control and monitor the given workload.
Many of the examples included with Optimizer Studio package demonstrate this approach.

Workload definition syntax explained

Below is a brief reference of the workload definition keywords:

  • kind: sync|async|accel - the workload invocation mode. Each kind is explained in more detail below.
  • run - the recipe for running the workload on each sample.
  • on_config_change - the preparational section, executed only on knob configuration change. The run section is still executed unconditionally. [optional]
  • command - the actual command executed. It can contain either direct shell commands or an external script call.
  • timeout - the sample will be invalidated in case the command does not complete within TIME (in HhMmSs format). When 0 or omitted, timeout is considered indefinite. [optional]
  • stop - the shell sequence used to stop the command. When omitted, kill -SIGKILL ${WORKLOAD_PID} will be used. [optional]

Sync workload (default)

The sync workload is the most often used kind of workload. The workload runs from the start to the end, and the target metric is sampled upon workload completion.

workload:
  kind: sync
  on_config_change:
    command: ./prepare.sh
    timeout: 1m30s
    stop: kill -SIGTERM ${WORKLOAD_PID}

  run:
    command: ./workload.sh
    timeout: 30s
    stop: kill -SIGTERM ${WORKLOAD_PID}

Async workload

The async workload is a workload that must be up and running while the target metric is sampled.
Oftentimes the async workload is useful in networking workload, as the network traffic stops upon workoad completion.
In this case, we request to sample the target metric TIME after launching the workload - sample_after: HhMmSs. The workload is stopped upon sampling the target metric.

workload:
  kind: async
  on_config_change:
    command: ./prepare.sh
    timeout: 1m30s
    stop: kill -SIGTERM ${WORKLOAD_PID}

  run:
    command: iperf3 -c <IP address> -t 0
    sample_after: 3s
    stop: killall iperf3

Accel workload

The accel workload is a workload run via accelerate utility. This workload is often executed in a production environment, and Optimizer Studio attempts a new knob configuration each time the workload restarts.

workload:
  kind: accel

Accessing knobs and metrics via {{}} templates

It is possible to get access to knob values and some metrics via {{}} (Jinja-style) templates, e.g.

domain:
  common:
    knobs:
      A: ...
      B: ...
      C: ...
  ...

workload:
  kind: sync
  command: |
    echo "{{A}} + {{B}} + {{C}}" | bc -l > /tmp/ans
  timeout: {{best.duration * 2}}

Each {{knob-name}} is substituted with the current knob value.
In order to avoid ambiguity, it is also possible to prefix the knob name with "knob.", e.g. {{knob.A}}.
Each metric name has to be prefixed by "baseline." or "best.".
Probably the most popular usecase is setting the timeout based on previous experiment duration, e.g. timeout: {{baseline.duration * 2}}.

Accessing multiple knobs at once

Oftentimes multiple knobs are required at once, e.g. when the knobs serve as compiler optimization flags:

workload:
  kind: sync
  on_config_change:
    command: |
      gcc {{get_knobs(".*")}} $STREAM.c -o $STREAM
  run:
    command: |
      $STREAM | awk '/Triad/ {print $2}' > /tmp/t_metric

The get_knobs(PATTERN) will be substituted by the values of all the knobs matching the pattern separated by a space.
The full format is get_knobs(PATTERN [, DELIMITER [, FORMAT]]), where

  • PATTERN - regular expression applied to the knob name
  • DELIMITER - a delimiter used to separate each two knob values [optional]
  • FORMAT - instead of substituting each knob with its value, we can employ a format recognizing %name and %value keywords. E.g. to export every knob as an environment variable, one could use: {{get_knobs(".*", "; ", "export %name=%value")}}. [optional]

Workload definition and multi-objective optimization (deprecated)

A workload is normally defined in a script supplied to Optimizer Studio via its command line.
Optimizer Studio also supports complex workload definition with scalarized muti-objective target.
The complex workload is defined in the configuration file as a sequence of steps, each comprising a script, metrics and validity checks, as following:

  1. Script: a script that runs the workload
  2. Metrics: after the script (if defined) completes, metrics are gathered from the filesystem
  3. Validity checks: after the metrics are gathered, they are validated in a sequence of tests. If all checks pass, the next workload step is executed. Otherwise, the knob configuration is deemed invalid and Optimizer Studio resumes testing a different configuration.

Upon completion and validation of all the steps, workload target metric is calculated. The workload target is specified through scalarization of the step metrics as described above.

Optimization target metric definition

The workload target can be set as target metric of the whole optimization.

Example

domain:
  common:

    ...

    target: workload.target:max

workload_settings:
  workloads:
    -
      script: ./workload_1.sh arg1
      metrics:
        workload1_output: /tmp/target1
        metric1: /tmp/metric1
      validity: 
        - workload1_output > 3200
        - metric1 < 200
    -
      script: ./workload_2.sh arg2
      metrics:
        workload2_output: /tmp/target2
      validity: workload2_output < 10000

  target: workload1_output * 0.12 + workload2_output * 8 - metric1

The above example demonstrates a 2-step complex workload.
Optimizer Studio comes up with a configuration to test.
It will then run ./workload_1.sh arg1. Upon completion, metrics will be sampled, and validity criteria applied - (workload1_output > 3200) && (metric1 < 200).
If both checks pass, Optimizer Studio will proceed to running the next step script (./workload_2.sh arg2). Otherwise, Optimizer Studio will come up with a different configuration, and start testing the first step again.

If all the workloads run correctly and pass their validity tests, Optimizer Studio will calculate the workload target according to the formula at the last line of the example.
Workload target is defined as optimization target metric (and is maximized).