Nevergrad Sweeper plugin

  

Nevergrad is a derivative-free optimization platform providing a library of state-of-the-art algorithms for hyperparameter search. This plugin provides Hydra applications a mechanism to use Nevergrad algorithms to optimize experiment/application parameters.

Installation

pip install hydra-nevergrad-sweeper --upgrade

Usage

Once installed, add hydra/sweeper=nevergrad to your command. Alternatively, override hydra/sweeper in your config:

defaults:
  - override hydra/sweeper: nevergrad

The default configuration is defined and documented here. There are several standard approaches for configuring plugins. Check this page for more information.

Example of training using Nevergrad hyperparameter search

We include an example of how to use this plugin. The file example/my_app.py implements an example of minimizing a (dummy) function using a mixture of continuous and discrete parameters.

You can discover the Nevergrad sweeper parameters with:

$ python your_app hydra/sweeper=nevergrad --cfg hydra -p hydra.sweeper

# @package hydra.sweeper
_target_: hydra_plugins.hydra_nevergrad_sweeper.nevergrad_sweeper.NevergradSweeper
optim:
  optimizer: NGOpt
  budget: 80
  num_workers: 10
  noisy: false
  maximize: false
  seed: null
  max_failure_rate: 0.0
params:
  db: choice(mnist, cifar)
  lr: {init: 0.02, step: 2.0, log: true}
  dropout: interval(0, 1)
  batch_size: int(interval(4, 16))

The function decorated with @hydra.main() returns a float which we want to minimize, the minimum is 0 and reached for:

db: mnist
lr: 0.12
dropout: 0.33
batch_size: 4

To run hyperparameter search and look for the best parameters for this function, clone the code and run the following command in the plugins/hydra_nevergrad_sweeper directory:

python example/my_app.py -m

You can also override the search space parametrization:

python example/my_app.py --multirun db=mnist,cifar batch_size=4,8,16 \
'lr=tag(log, interval(0.001, 1))' 'dropout=interval(0,1)'

The initialization of the sweep and the first 5 evaluations (out of 100) look like this:

[2020-10-08 20:13:53,592][HYDRA] NevergradSweeper(optimizer=NGOpt, budget=100, num_workers=10) minimization
[2020-10-08 20:13:53,593][HYDRA] with parametrization Dict(batch_size=Choice(choices=Tuple(4,8,16),weights=Array{(1,3)}),db=Choice(choices=Tuple(mnist,cifar),weights=Array{(1,2)}),dropout=Scalar{Cl(0,1,b)}[sigma=Log{exp=2.0}],lr=Log{exp=3.162277660168379,Cl(0.001,1,b)}):{'db': 'mnist', 'lr': 0.03162277660168379, 'dropout': 0.5, 'batch_size': 8}
[2020-10-08 20:13:53,593][HYDRA] Sweep output dir: multirun/2020-10-08/20-13-53
[2020-10-08 20:13:55,023][HYDRA] Launching 10 jobs locally
[2020-10-08 20:13:55,023][HYDRA]        #0 : db=mnist lr=0.03162277660168379 dropout=0.5 batch_size=16
[2020-10-08 20:13:55,217][__main__][INFO] - dummy_training(dropout=0.500, lr=0.032, db=mnist, batch_size=16) = 13.258
[2020-10-08 20:13:55,218][HYDRA]        #1 : db=cifar lr=0.018178519762066934 dropout=0.5061074452336254 batch_size=4
[2020-10-08 20:13:55,408][__main__][INFO] - dummy_training(dropout=0.506, lr=0.018, db=cifar, batch_size=4) = 0.278
[2020-10-08 20:13:55,409][HYDRA]        #2 : db=cifar lr=0.10056825918734161 dropout=0.6399687427725211 batch_size=4
[2020-10-08 20:13:55,595][__main__][INFO] - dummy_training(dropout=0.640, lr=0.101, db=cifar, batch_size=4) = 0.329
[2020-10-08 20:13:55,596][HYDRA]        #3 : db=mnist lr=0.06617542958182834 dropout=0.5059497416026679 batch_size=8
[2020-10-08 20:13:55,812][__main__][INFO] - dummy_training(dropout=0.506, lr=0.066, db=mnist, batch_size=8) = 5.230
[2020-10-08 20:13:55,813][HYDRA]        #4 : db=mnist lr=0.16717013388679514 dropout=0.6519070394318255 batch_size=4
...
[2020-10-08 20:14:27,988][HYDRA] Best parameters: db=cifar lr=0.11961221693764439 dropout=0.37285878409770895 batch_size=4

and the final 2 evaluations look like this:

[HYDRA] 	#8 : db=mnist batch_size=4 lr=0.094 dropout=0.381
[__main__][INFO] - my_app.py(dropout=0.381, lr=0.094, db=mnist, batch_size=4) = 1.077
[HYDRA] 	#9 : db=mnist batch_size=4 lr=0.094 dropout=0.381
[__main__][INFO] - my_app.py(dropout=0.381, lr=0.094, db=mnist, batch_size=4) = 1.077
[HYDRA] Best parameters: db=mnist batch_size=4 lr=0.094 dropout=0.381

The run also creates an optimization_results.yaml file in your sweep folder with the best evaluated parameters found during the sweep:

best_evaluated_result: 0.381

best_evaluated_params:
  batch_size: 4
  db: mnist
  dropout: 0.381
  lr: 0.094

name: nevergrad

Defining the parameters

The plugin supports two types of parameters: Choices and Scalars. They can be defined either through a config file or command-line override.

Defining through command-line override

Hydra provides an override parser that supports rich syntax. More documentation can be found in (Override Grammar/Basic) and (Override Grammar/Extended). We recommend you go through them first before proceeding with this doc.

Choices

To override a field with choices:

'key=1,5'
'key=shuffle(range(1, 8))'
'key=range(1,5)'

You can tag an override with ordered to indicate it's a TransitionChoice

key=tag(ordered, choice(1,2,3))

Scalar

key=interval(1,12)             # Intervals are floats by default
key=int(interval(1,8))         # Scalar bounds are cast to an int
key=tag(log, interval(1,12))   # call ng.p.Log if tagged with log
key={init: 0.02, step: 2.0, log: true}  # Unbounded log-scale scalar

Defining through config file

The search space can be defined under hydra.sweeper.params. The definition is consistent with the equivalent command-line override.

warning

hydra.sweeper.parametrization is deprecated in Hydra 1.4 and will be removed in Hydra 1.5. Use hydra.sweeper.params instead.

Choices

Choices use the same syntax as command-line overrides. For example, translate key=shuffle(range(1, 8)) to key: shuffle(range(1, 8)) in the config file.

Fixed strings containing override grammar characters must retain quotes as part of the configured value. For example, use label: "'a,b'" to configure the literal string a,b; label: a,b defines a choice sweep.

Scalars

Scalar definitions also support the override syntax shown above. The following keys can be passed in a dictionary to define unbounded scalars:

• init: optional initial value
• lower : optional lower bound
• upper: optional upper bound
• log: set to true for log distributed values
• step: optional step size for looking for better parameters. In linear mode, this is an additive step; in logarithmic mode it is multiplicative.
• integer: set to true for integers (favor floats over integers whenever possible)

Providing only lower and upper bound will set the initial value to the middle of the range and the step to a sixth of the range.