Skip to main content

Hydra 1.0

· 8 min read
Omry Yadan

logo

After many months and a lot of hard work by many people, Hydra 1.0 is finally out!
Despite some very big changes, this is still the Hydra you love - only bigger, stronger and with more heads.

Major new features in Hydra 1.0​

  • Config type safety via Structured Configs
  • More powerful command line
  • New plugins enabling remote launching and hyper parameter optimization
  • Improved error reporting
  • Reduce nesting levels with config packages

Config type safety via Structured Configs​

Structured Configs is a powerful new feature that enables strongly typed configs that can be validated both statically and at runtime. With Structured Configs, you use Python dataclasses to describe your configuration structure and types. They can be used as an alternative to YAML files, or as a way to validate YAML files automatically.

See example (Click to expand)

This example is using a Structured Config as an alternative to a configuration file.

@dataclass
class MySQLConfig:
    host: str = "localhost"
    port: int = 3306

cs = ConfigStore.instance()
cs.store(name="config", node=MySQLConfig)

@hydra.main(config_name="config")
def my_app(cfg: MySQLConfig) -> None:
    if cfg.pork == 80: # pork should be port!
        print("Is this a webserver?!")

Duck-typing the config object as MySQLConfig enables static type checkers like mypy to catch type errors before you run your code:

$ mypy my_app.py
my_app.py:22: error: "MySQLConfig" has no attribute "pork"
Found 1 error in 1 file (checked 1 source file)

Hydra will catch typos, or type errors in the command line:

$ python my_app.py port=fail
Error merging override port=fail
Value 'fail' could not be converted to Integer
        full_key: port
        reference_type=Optional[MySQLConfig]
        object_type=MySQLConfig

Learn more at the Structured Configs tutorial.

More powerful command line​

Hydra 1.0 introduces a new command line with many new capabilities, such as:

  • Override, Add or Delete config values or Default list choices
  • Cast values to coerce their type
  • Specify dictionaries and lists as values
See examples
Override as usual



Prefix with + to add a new field to
the config



Prefix with ~ to delete a field from
the config

Cast values to coerce their type.



Support for dictionaries and lists







$ python my_app.py db.user=root
db:
  user: root
  pass: secret
$ python my_app.py +db.timeout=10
db:
  user: omry
  pass: secret
  timeout: 10
$ python my_app.py '~db.pass'
db:
  user: omry
$ python my_app.py 'db.pass=str(42)'
db:
  user: omry
  pass: '42'
$ python my_app.py \
  '+db.params={a:10,b:20}'
db:
  user: omry
  pass: secret
  params:
    a: 10
    b: 20

Learn more at the Basic Override syntax page.

Sweeper improvements​

Advanced command line capabilities are making Hydra's Basic Sweeper more powerful.

  • Specify numeric ranges via the command line
  • Use glob to select multiple config group options without specifying them explicitly
  • Control sweeping order with sort and shuffle
See examples
Use glob to easily select
multiple config group options
Use range to specify a range of 
numbers


You can sort sweep to run order






You can also shuffle sweeps to run
in random order

$ python my_app.py -m 'db=glob(*)'
# Will sweep over all db options
$ python my_app.py --multirun \
  '+number=range(1,4)'
# number will take the values
# 1,2 and 3 
$ python my_app.py --multirun \
  '+num=sort(3,1,2)'
# Sweep over num in order

$ python my_app.py --multirun \
  '+num=sort(3,1,2,reverse=true)'
# Sweep over num in reverse order
$ python my_app.py --multirun \
  '+num=shuffle(3,1,2)'
# Sweep over num in random order

Learn more at the Extended Override grammar page.

New plugins​

Launchers​

One of the early promises of Hydra was that it will enable you to easily launch your application to different clusters. Hydra 1.0 adds a few Launchers plugins that starts to make good on that promise.

Sweepers​

Two new Sweeper plugins enables you to automatically find optimal parameters without changing a line of code.

Note that both Sweepers are still in beta and some changes are expected soon. Your feedback can help shape them.

Tab completion​

In addition to Bash, Hydra now supports Fish shell Tab Completion.

Compose API improvements​

The experimental Compose API is maturing. It is now possible to initialize Hydra in one of 3 ways:

  • initialize(): Initialize with a config path relative to the caller
  • initialize_config_module() : Initialize with config_module (absolute)
  • initialize_config_dir() : Initialize with a config_dir on the file system (absolute)

All initialization methods can be used to initialize Hydra globally or in a context. Making the Compose API ideal for Unit tests and Jupyter notebooks.

Learn more at the Compose API page.

Improved error reporting​

Reported errors will have un-interesting stack frames removed by default. You can enable the complete stack trace with the environment variable HYDRA_FULL_ERROR=1.

See example of an error
@hydra.main()
def my_app(cfg: DictConfig) -> None:
    1 / 0 # hmmm, seems fishy

if __name__ == "__main__":
    my_app()
$ python my_app.py
Traceback (most recent call last):
  File "my_app.py", line 9, in my_app
    1 / 0
ZeroDivisionError: division by zero

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

See example of a complete stack trace
$ HYDRA_FULL_ERROR=1 python my_app.py
Traceback (most recent call last):
  File "my_app.py", line 13, in <module>
    my_app()
  File "/home/omry/dev/hydra/hydra/main.py", line 32, in decorated_main
    _run_hydra(
  File "/home/omry/dev/hydra/hydra/_internal/utils.py", line 355, in _run_hydra
    run_and_report(
  File "/home/omry/dev/hydra/hydra/_internal/utils.py", line 210, in run_and_report
    raise ex
  File "/home/omry/dev/hydra/hydra/_internal/utils.py", line 207, in run_and_report
    return func()
  File "/home/omry/dev/hydra/hydra/_internal/utils.py", line 356, in <lambda>
    lambda: hydra.run(
  File "/home/omry/dev/hydra/hydra/_internal/hydra.py", line 107, in run
    return run_job(
  File "/home/omry/dev/hydra/hydra/core/utils.py", line 125, in run_job
    ret.return_value = task_function(task_cfg)
  File "my_app.py", line 9, in my_app
    1 / 0
ZeroDivisionError: division by zero

Reduce nesting levels with config packages​

Hydra 1.0 introduces the ability to specify the root config package in the config file. Specifying the root config package help reducing nesting levels in config files. Config packages can be overridden via the Defaults List or the command line, allowing the reuse of the the same config in multiple place in the resulting output without having to duplicate it.

See example of reduced nesting

The following two db/mysql.yaml files are equivalent:

Hydra 0.11
db:
  host: localhost
  port: 3306
Hydra 1.0
# @package _group_
host: localhost
port: 3306
See example of config reuse
Add the `mysql` config in the packages `db.src` and `db.dst`, reusing `db/mysql.yaml`.
config.yaml
defaults:
 - [email protected]: mysql
 - [email protected]: mysql




Interpretation
db:
  src:
    host: localhost
    port: 3306
  dst:
    host: localhost
    port: 3306

Learn more at the Overriding packages page.

Misc improvements​

  • The params field is eliminated from instantiated objects configs (details)
  • Support for setting environment variables via the config. (hydra.job.env_set) (details)
  • Hydra's config can now be accessed through interpolation using ${hydra:key}, for example ${hydra:job.name} (details)
  • Introduced --info flag for quick access to debug information (details)
  • Add --package flag, can be used with --cfg to print a specific config package (details)
  • Override the config_path and config_name from the command line with --config-name and --config-path (details)
  • Add an additional config directory to the search path with --config-dir (details)

Migrating from 0.11​

For most people, migrating from 0.11 to 1.0 will be smooth. However, you will have issues if you are relying on Python 2 support or have used internal APIs.

  • Hydra 1.0 drops support for Python 2.
  • If you are relying on internal APIs, Hydra 1.0 will likely break your code. Maintaining backward compatibility for internal APIs is not a goal.
  • Hydra 0.11 Configs pickled and stored will not unpickle with Hydra 1.0 due to internal changes in OmegaConf.

There are multiple new deprecation warnings, each with a link to mini migration guide.

That's it for now, take Hydra 1.0 for a spin!