Hydra 1.0

Omry Yadan

Omry Yadan

Creator of Hydra

logo

After many months and a lot of hard work by many people, Hydra 1.0 is finally out!
Despite some very big changes, this is still the Hydra you love - only bigger, stronger and with more heads.

Major new features in Hydra 1.0

  • Config type safety via Structured Configs
  • More powerful command line
  • New plugins enabling remote launching and hyper parameter optimization
  • Improved error reporting
  • Reduce nesting levels with config packages

Config type safety via Structured Configs

Structured Configs is a powerful new feature that enables strongly typed configs that can be validated both statically and at runtime. With Structured Configs, you use Python dataclasses to describe your configuration structure and types. They can be used as an alternative to YAML files, or as a way to validate YAML files automatically.

See example (Click to expand)

This example is using a Structured Config as an alternative to a configuration file.

@dataclass
class MySQLConfig:
host: str = "localhost"
port: int = 3306
cs = ConfigStore.instance()
cs.store(name="config", node=MySQLConfig)
@hydra.main(config_name="config")
def my_app(cfg: MySQLConfig) -> None:
if cfg.pork == 80: # pork should be port!
print("Is this a webserver?!")

Duck-typing the config object as MySQLConfig enables static type checkers like mypy to catch type errors before you run your code:

$ mypy my_app.py
my_app.py:22: error: "MySQLConfig" has no attribute "pork"
Found 1 error in 1 file (checked 1 source file)

Hydra will catch typos, or type errors in the command line:

$ python my_app.py port=fail
Error merging override port=fail
Value 'fail' could not be converted to Integer
full_key: port
reference_type=Optional[MySQLConfig]
object_type=MySQLConfig

Learn more at the Structured Configs tutorial.

More powerful command line

Hydra 1.0 introduces a new command line with many new capabilities, such as:

  • Override, Add or Delete config values or Default list choices
  • Cast values to coerce their type
  • Specify dictionaries and lists as values
See examples
Override as usual
Prefix with + to add a new field to
the config
Prefix with ~ to delete a field from
the config
Cast values to coerce their type.
Support for dictionaries and lists
$ python my_app.py db.user=root
db:
user: root
pass: secret
$ python my_app.py +db.timeout=10
db:
user: omry
pass: secret
timeout: 10
$ python my_app.py '~db.pass'
db:
user: omry
$ python my_app.py 'db.pass=str(42)'
db:
user: omry
pass: '42'
$ python my_app.py \
'+db.params={a:10,b:20}'
db:
user: omry
pass: secret
params:
a: 10
b: 20

Learn more at the Basic Override syntax page.

Sweeper improvements

Advanced command line capabilities are making Hydra's Basic Sweeper more powerful.

  • Specify numeric ranges via the command line
  • Use glob to select multiple config group options without specifying them explicitly
  • Control sweeping order with sort and shuffle
See examples
Use glob to easily select
multiple config group options
Use range to specify a range of
numbers
You can sort sweep to run order
You can also shuffle sweeps to run
in random order
$ python my_app.py -m 'db=glob(*)'
# Will sweep over all db options
$ python my_app.py --multirun \
'+number=range(1,4)'
# number will take the values
# 1,2 and 3
$ python my_app.py --multirun \
'+num=sort(3,1,2)'
# Sweep over num in order
$ python my_app.py --multirun \
'+num=sort(3,1,2,reverse=true)'
# Sweep over num in reverse order
$ python my_app.py --multirun \
'+num=shuffle(3,1,2)'
# Sweep over num in random order

Learn more at the Extended Override grammar page.

New plugins

Launchers

One of the early promises of Hydra was that it will enable you to easily launch your application to different clusters. Hydra 1.0 adds a few Launchers plugins that starts to make good on that promise.

Sweepers

Two new Sweeper plugins enables you to automatically find optimal parameters without changing a line of code.

Note that both Sweepers are still in beta and some changes are expected soon. Your feedback can help shape them.

Tab completion

In addition to Bash, Hydra now supports Fish shell Tab Completion.

Compose API improvements

The experimental Compose API is maturing. It is now possible to initialize Hydra in one of 3 ways:

  • initialize(): Initialize with a config path relative to the caller
  • initialize_config_module() : Initialize with config_module (absolute)
  • initialize_config_dir() : Initialize with a config_dir on the file system (absolute)

All initialization methods can be used to initialize Hydra globally or in a context. Making the Compose API ideal for Unit tests and Jupyter notebooks.

Learn more at the Compose API page.

Improved error reporting

Reported errors will have un-interesting stack frames removed by default. You can enable the complete stack trace with the environment variable HYDRA_FULL_ERROR=1.

See example of an error
@hydra.main()
def my_app(cfg: DictConfig) -> None:
1 / 0 # hmmm, seems fishy
if __name__ == "__main__":
my_app()
$ python my_app.py
Traceback (most recent call last):
File "my_app.py", line 9, in my_app
1 / 0
ZeroDivisionError: division by zero
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

See example of a complete stack trace
$ HYDRA_FULL_ERROR=1 python my_app.py
Traceback (most recent call last):
File "my_app.py", line 13, in <module>
my_app()
File "/home/omry/dev/hydra/hydra/main.py", line 32, in decorated_main
_run_hydra(
File "/home/omry/dev/hydra/hydra/_internal/utils.py", line 355, in _run_hydra
run_and_report(
File "/home/omry/dev/hydra/hydra/_internal/utils.py", line 210, in run_and_report
raise ex
File "/home/omry/dev/hydra/hydra/_internal/utils.py", line 207, in run_and_report
return func()
File "/home/omry/dev/hydra/hydra/_internal/utils.py", line 356, in <lambda>
lambda: hydra.run(
File "/home/omry/dev/hydra/hydra/_internal/hydra.py", line 107, in run
return run_job(
File "/home/omry/dev/hydra/hydra/core/utils.py", line 125, in run_job
ret.return_value = task_function(task_cfg)
File "my_app.py", line 9, in my_app
1 / 0
ZeroDivisionError: division by zero

Reduce nesting levels with config packages

Hydra 1.0 introduces the ability to specify the root config package in the config file. Specifying the root config package help reducing nesting levels in config files. Config packages can be overridden via the Defaults List or the command line, allowing the reuse of the the same config in multiple place in the resulting output without having to duplicate it.

See example of reduced nesting

The following two db/mysql.yaml files are equivalent:

Hydra 0.11
db:
host: localhost
port: 3306
Hydra 1.0
# @package _group_
host: localhost
port: 3306
See example of config reuseAdd the `mysql` config in the packages `db.src` and `db.dst`, reusing `db/mysql.yaml`.
config.yaml
defaults:
Interpretation
db:
src:
host: localhost
port: 3306
dst:
host: localhost
port: 3306

Learn more at the Overriding packages page.

Misc improvements

  • The params field is eliminated from instantiated objects configs (details)
  • Support for setting environment variables via the config. (hydra.job.env_set) (details)
  • Hydra's config can now be accessed through interpolation using ${hydra:key}, for example ${hydra:job.name} (details)
  • Introduced --info flag for quick access to debug information (details)
  • Add --package flag, can be used with --cfg to print a specific config package (details)
  • Override the config_path and config_name from the command line with --config-name and --config-path (details)
  • Add an additional config directory to the search path with --config-dir (details)

Migrating from 0.11

For most people, migrating from 0.11 to 1.0 will be smooth. However, you will have issues if you are relying on Python 2 support or have used internal APIs.

  • Hydra 1.0 drops support for Python 2.
  • If you are relying on internal APIs, Hydra 1.0 will likely break your code. Maintaining backward compatibility for internal APIs is not a goal.
  • Hydra 0.11 Configs pickled and stored will not unpickle with Hydra 1.0 due to internal changes in OmegaConf.

There are multiple new deprecation warnings, each with a link to mini migration guide.

That's it for now, take Hydra 1.0 for a spin!