Experiment Workflow¶
Overview¶
The Experiment Workflow consists of several stages, which can be selected via --stages
command line switch,
where the desired stages can be listed, separated by comma, e.g. --stages optimization,refinement
.
The available workflow stages are:
- readiness
- optimization [default]
- refinement
- validation
The stages are pre-ordered irrespective of the order of appearance on the command line, yet the user is free to select which stages should be executed. Some stages require additional configuration parameters in the knob file. Some stages can be selected only when permitted by the license.
It is possible to revisit the workflow of an ongoing experiment. Stop the experiment, then restart it
(with --retain
command line switch) with only stages of interest selected.
Readiness Stage¶
The Readines Stage purpose is finding out the correct optimization parameters, before running the optimization process.
Optimizer Studio runs the workload several times using baseline knob configuration, and finds out the approximate time of running a workload, as well as statistical characteristics of the target metric for the baseline configuration - mean value, standard deviation, coefficient of variation (CV), etc.
After this stage, the user can set optimization parameters with more confidence.
The results of the Readiness stage are reported to Experiment Management System.
Readiness Stage Configuration¶
The following section of a knob file is used for configuration:
stages:
readiness:
min_samples: 10
duration: 15m
The Readiness stage will run until min_samples
are acquired or duration
has passed, the earlier.
duration
values should have the syntax: AhBmCs
, where A, B, C
are the values, and h, m, s
mean "hours", "minutes" or "seconds" correspondingly.
For example: 1h30m
, 15m
, etc.
Optimization Stage¶
Optimization stage comprises the main optimization process by running the workload and testing different knob configurations. This is the main and default functionality of Optimizer Studio.
Optimization Stage Configuration¶
The following section of a knob file is used for configuration:
stages:
optimization:
min_configs: 30
timeout: 15m
configs_without_improvement: 10
Or
stages:
optimization:
duration: 15m
The Optimization stage will exercise min_configs
(if provided) no matter the progress, but not pass the timeout
(if provided).
After testing N configurations, the Optimization stage will run to completion (progress = 100%) or timeout
(if provided).
If configs_without_improvement
provided, the Optimization stage is considered completed if no improvement is observed for M consecutive configurations or progress reaches 100%.
duration
is incompatible with other options, and sets the precise optimization duration.
Refinement Stage¶
The Refinement Stage runs after the Optimization Stage completes.
The Refinement is applied to the best configuration dicovered throughout the previous stages, and strives to minimize the number of knobs which differ from their baseline values.
It is possible to stop Optimizer Studio, and restart it (with --retain
command line switch) with only refinement
stage selected.
In this case, Optimizer Studio will proceed to refining the best configuration dicovered till this point.
Refinement Stage Configuration¶
The Refinement stage can be configured via the knob file:
stages:
refinement:
std_tolerance: N # default: 1.0
improv_tolerance: P # default: 0.01%
The purpose of the Refinement stage is to identify (and return to baseline) the knobs that do not contribute to the improvement. However, in a noisy environment it is difficult to tell a small improvement from a random noise. The parameters above are used to set this sensitivity limit:
std_tolerance
- two measurements are considered distinct if the difference between them is more than N times the standard deviationimprov_tolerance
- an improvement below P% is considered noise
In a completely quiet environment, it makes sense to set improv_tolerance: 0
.
Validation Stage¶
The purpose of the Validation stage is to assess the statistical significance of the best performance
achieved throughout the optimization process.
The best configuration is attempted N times while statistical information is collected, such as
mean value, standard deviation, coefficient of variation (CV).
In case there are not enough baseline samples for the statistical significance assessment, the additional
baseline samples will be made.
Validation Stage Configuration¶
The following section of a knob file is used for configuration:
stages:
validation:
min_samples: 10
duration: 15m