Configuration

Use these configuration fields to customize your sweep.

Top-level key

Meaning

name

The name of the sweep displayed in the W&B UI

description

Text description of the sweep

program

Training script (required)

metric

Specify the metric to optimize (used by some search strategies and stopping criteria)

method

Specify the search strategy (required)

early_terminate

Specify the stopping critera

parameters

Specify parameters bounds to search (required)

Metric

Specify the metric to optimize. This metric should be logged by your training script.

metric sub-key

Meaning

name

Name of the metric to optimize

goal

minimize or maximize (Default is minimize)

Examples

Maximize
Minimize
metric:
name: val_loss
goal: maximize
metric:
name: val_loss

Search Strategy

Specify the search strategy with the method key in the sweep configuration file.

method

Meaning

grid

Grid Search iterates over all possible sets of parameter values.

random

Random Search chooses random sets of values.

bayes

Bayesian Optimization uses a gaussian process to model the function and then chooses parameters to optimize probability of improvement. This strategy requires a metric key to be specified.

Examples

Random search
Grid search
Bayes search
method: random
method: grid
method: bayes
metric:
name: val_loss
goal: minimize

Stopping Criteria

Early Termination speeds up hyperparameter search by killing off pooly performing runs.

early_terminate sub-key

Meaning

type

specify the stopping algorithm

We support these stopping algorithms:

type

Meaning

hyperband

Use the hyperband method

envelope

Use an envelope method for early termination

Hyperband parameters

Hyperband stopping evaluates whether a program should be stopped or permitted to continue at one or more brackets during the execution of the program. Brackets are configured at static iterations for a specified metric (where an iteration is the number of times a metric has been logged -- if the metric is logged every epoch, then there are epoch iterations).

In order to specify the bracket schedule eithermin_iter or max_iter needs to be defined.

early_terminate sub-key

Meaning

min_iter

specify the iteration for the first bracket

max_iter

specify the maximum number of iterations for the program

s

specify the total number of brackets (required for max_iter)

eta

specify the bracket multiplier schedule (default: 3)

Examples

Hyperband (min_iter)
Hyperband (max_iter)
Envelope
early_terminate:
type: hyperband
min_iter: 3

Brackets: 3, 9 (3*eta), 27 (9 * eta), 81 (27 * eta)

early_terminate:
type: hyperband
max_iter: 27
s: 2

Brackets: 9 (27/eta), 3 (9/eta)

early_terminate:
type: envelope

Parameters

The parameters dictionary specifies the ranges of configuration parameters.

Values

Meaning

distribution: (distribution)

A distribution from the distrbution table below. If not specified, the sweep will set to uniform if max and min are set, categorical if values are set and constant if value is set.

min: (float) max: (float)

Continuous values between min and max

min: (int) max: (int)

Integers between min and max

values: [(float), (float), ...]

Discrete values

value: (float)

A constant

mu: (float)

Mean for normal or lognormal distributions

sigma: (float)

Standard deviation for normal or lognormal distributions

q: (float)

Quantization parameter for quantized distributions

Distributions

Name

Meaning

constant

Constant distribution. Must specify value.

categorical

Categorical distribution. Must specify values.

int_uniform

Uniform integer. Must specify max and min as integers.

uniform

Uniform continuous. Must specify max and min as floats.

q_uniform

Quantized uniform. Returns round(X / q) * q where X is uniform. Q defaults to 1.

log_uniform

Log uniform. Number between exp(min) and exp(max) so that the logarithm of the return value is uniformly distributed.

q_log_uniform

Quantized log uniform. Returns round(X / q) * q where X is log_uniform. Q defaults to 1.

normal

Normal distribution. Value is chosen from normal distribution. Can set mean mu (default 0) and std dev sigma (default 1).

q_normal

Quantized normal distribution. Returns round(X / q) * q where X is normal. Q defaults to 1.

log_normal

Log normal distribution. Value is chosen from log normal distribution. Can set mean mu (default 0) and std dev sigma (default 1).

q_log_normal

Quantized log normal distribution. Returns round(X / q) * q where X is log_normal. Q defaults to 1.

Example

parameters:
my-parameter:
min: 1
max: 20