Nevergrad is a derivative-free optimization platform proposing a library of state-of-the art algorithms for hyperparameter search. This plugin provides a mechanism for Hydra applications to use Nevergrad algorithms for the optimization of experiments/applications parameters.
Once installed, add
hydra/sweeper=nevergrad to your command command. Alternatively, override
hydra/sweeper in your config:
The default configuration is here.
Example of training using Nevergrad hyperparameter search
We include an example of how to use this plugin. The file
example/dummy_training.py implements an example of how to perform minimization of a (dummy) function including a mixture of continuous and discrete parameters.
You can discover the Nevergrad sweeper parameters with:
The function decorated with
@hydra.main() returns a float which we want to minimize, the minimum is 0 and reached for:
To run hyperparameter search and look for the best parameters for this function, clone the code and run the following command in the
You can also override the search space parametrization:
The initialization of the sweep and the first 5 evaluations (out of 100) look like this:
and the final 2 evaluations look like this:
The run also creates an
optimization_results.yaml file in your sweep folder with the parameters recommended by the optimizer:
Defining the parameters
The plugin can use 2 types of parameters:
Choices are defined with comma-separated values in the command-line (
batch_size=4,8,12,16) or with a list in a config file.
By default, values are processed as floats if all can be converted to it, but you can modify this behavior by adding colon-separated specifications
str before the the list. (eg.:
Note: sequences of increasing scalars are treated as a special case, easier to solve. Make sure to specify it this way when possible.
Scalars can be defined:
through a commandline override with
:-separated values defining a range (eg:
dropout=0:1). You can add specifications for log distributed values (eg.:
lr=log:0.001:1) or integer values (eg.:
batch_size=int:4:8) or a combination of both (eg.:
through a config files, with fields:
init: optional initial value
lower: optional lower bound
upper: optional upper bound
log: set to
truefor log distributed values
step: optional step size for looking for better parameters. In linear mode this is an additive step, in logarithmic mode it is multiplicative.
integer: set to
truefor integers (favor floats over integers whenever possible)
upperbound will set the initial value to the middle of the range, and the step to a sixth of the range.
Note: unbounded scalars (scalars with no upper and/or lower bounds) can only be defined through a config file.