Parameters¶
See also
For the technical specifications, go to valohai.yaml step.parameters section.
The Wikipedia definition of a hyperparameter is: “a parameter whose value is set before the learning process begins.” This definition exists to differentiate these values that control how the machine learning is conducted from parameters such as neural network weights that are actively changed during training. Thus, hyperparameters are a subset of parameters.
In the context of Valohai, all parameters to be recorded are defined before launching an execution. If you wish to record something that is defined or changes during runtime, use Valohai metadata.
Note
Valohai supports numerous workloads in addition to normal machine learning training such as data generation and feature extraction. Check out What is Valohai? page to learn what else could be achieved.
This is the reason why we don’t always use the training-centric hyperparameter term.
Defining parameters¶
To keep track of parameters in Valohai, you add them as part of your YAML step.parameters.
Example of a valohai.yaml
with a step defining 3 parameters:
- step:
name: train
image: python:3.6
command:
- python train.py {parameters}
parameters:
- name: max_steps
pass-as: --max_steps={v}
type: float
default: 300
- name: learning_rate
pass-as: --learning_rate={v}
type: float
default: 0.001
- name: dropout
pass-as: --dropout={v}
type: float
default: 0.9
The above would generate the following command by default:
python train.py --max_steps=300 --learning_rate=0.001 --dropout=0.9
You can then parse the command-line arguments in your script.
import argparse
def parse_args():
parser = argparse.ArgumentParser()
parser.add_argument('--max_steps', type=int, default=300)
parser.add_argument('--learning_rate', type=float, default=0.001)
parser.add_argument('--dropout', type=float, default=0.9)
return parser.parse_args()
args = parse_args()
print("Max steps: %s" % args.max_steps)
valohai-utils
Python users can use the valohai-utils helper library to define and parse parameters.
import valohai
my_parameters = {
'max_steps': 300,
'learning_rate': 0.001,
'dropout': 0.9,
}
valohai.prepare(step="helloworld", default_parameters=my_parameters)
print("Max steps: %s" % valohai.parameters('max_steps').value)
and generate valohai.yaml
configuration file with the parameter definitions:
vh yaml step helloworld.py
vh exec run helloworld --adhoc
Selecting values¶
When you are creating a new Valohai execution using the web user interface, look for the Parameters
subsection at the bottom.

Default values of the parameters are defined by the valohai.yaml
, but they can be tweaked in
the web user interface, command-line client or the API.
All changes are version controlled as part of a Valohai execution.
Hyperparameter search¶
It can be daunting to try different hyperparameters one-by-one. Valohai offers a mechanism to do hyperparameter searches using parallel executions and a grid search. These are called tasks.
When starting a task, instead of a single value for a single hyperparameter, you get to define multiple values at once. There are various modes to choose from e.g. Single, Multiple, Linear, Logspace and Random.
Single¶

Single means just a single value for a hyperparameter you do not want to search for.
Multiple¶

Multiple means a list of all the values to try for a specific hyperparameter. For example, here we are trying out 4 different values (0.81, 0.84, 0.86 and 0.91).
Logspace¶

Logspace is a search with values inside a specific range in logarithmic space. For example, here we are trying out 4 different values between 2^2 - 2^8 (base^start - base^end)
Random¶

Finally, if you want to “gamble”, you can search randomly with a specified range and distribution. For example, here we are trying out 10 different random values between 0.001 and 0.002.
Grid search¶

When we are searching for multiple values for multiple hyperparameters, all permutations are searched. This is also called grid search.
For example, here we have 6 different values for learning_rate and 6 different values for dropout. In total this is 6*6 = 36 executions. Valohai calculates the number for permutations for you and you can see it in the right bottom corner of this screenshot.
Bayesian optimization¶
Using interactive hyperparameter optimisation can make hyperparameter tuning faster and more efficient than for example using a random search or an exhaustive grid search.
Valohai uses the open source Hyperopt-library’s Tree Parzen Estimator algorithm to use the hyperparameters and outputs of the previous executions to suggest future execution hyperparameters.
Under the hood, Bayesian optimization (of which TPE is an implementation) works in the following steps:
Create startup executions using random search
Based on these executions, create a simplified function to model the relationship between the hyperparameters and the target metric value (for example “loss”)
Based on this simplification of their relationship, find the optimal values for the hyper parameter to make the target metric as close to the target value as possible
Run the next batch of executions and repeat the process from step 2.
