Skip to main content
Version: Next

Basic Override syntax​

You can manipulate your configuration with overrides (via the command line or the Compose API). This includes:

  • Modifying the Defaults List
  • Modifying the config object

Overrides matching a config group are modifying the Defaults List; The rest are manipulating the config object.

Basic examples​

Modifying the Config Object​

  • Overriding a config value : foo.bar=value
  • Appending a config value : +foo.bar=value
  • Appending or overriding a config value : ++foo.bar=value
  • Removing a config value : ~foo.bar, ~foo.bar=value

Modifying the Defaults List​

  • Overriding selected Option: db=mysql, server/db=mysql
  • Appending to Defaults List: +db=mysql, +server/db=mysql
  • Deleting from Defaults List: ~db, ~db=mysql, ~server/db, ~server/db=mysql

Grammar​

Hydra supports a rich DSL in the command line. Below are the parser rules from grammar. You can see the full Lexer and Parser definitions on GitHub.

OverrideParser.g4
// High-level command-line override.

override: (
key EQUAL value? // key=value, key= (for empty value)
| TILDE key (EQUAL value?)? // ~key | ~key=value
| PLUS PLUS? key EQUAL value? // +key= | +key=value | ++key=value
) EOF;

// Key:
key : packageOrGroup (AT package)?; // key | group@pkg

packageOrGroup: package | ID (SLASH ID)+; // db, hydra/launcher
package: ( | ID | KEY_SPECIAL | DOT_PATH); // db, $db, hydra.launcher, or the empty (for _global_ package)

// Elements (that may be swept over).

value: element | simpleChoiceSweep;

element:
primitive
| listContainer
| dictContainer
| function
;

simpleChoiceSweep:
element (COMMA element)+ // value1,value2,value3
;

// Functions.

argName: ID EQUAL;
function: ID POPEN (argName? element (COMMA argName? element )* )? PCLOSE;

// Data structures.

listContainer: BRACKET_OPEN // [], [1,2,3], [a,b,[1,2]]
(element(COMMA element)*)?
BRACKET_CLOSE;

dictContainer: BRACE_OPEN (dictKeyValuePair (COMMA dictKeyValuePair)*)? BRACE_CLOSE; // {}, {a:10,b:20}
dictKeyValuePair: dictKey COLON element;

// Primitive types.

primitive:
QUOTED_VALUE // 'hello world', "hello world"
| ( ID // foo-bar_10
| NULL // null, NULL
| INT // 0, 10, -20, 1_000_000
| FLOAT // 3.14, -20.0, 1e-1, -10e3
| BOOL // true, TrUe, false, False
| INTERPOLATION // ${foo.bar}, ${oc.env:USER,me}
| UNQUOTED_CHAR // /, -, \, +, ., $, %, *, @, ?, |
| COLON // :
| ESC // \\, \(, \), \[, \], \{, \}, \:, \=, \ , \\t, \,
| WS // whitespaces
)+;

// Same as `primitive` except that `COLON` and `INTERPOLATION` are not allowed.
dictKey:
( ID // foo-bar_10
| NULL // null, NULL
| INT // 0, 10, -20, 1_000_000
| FLOAT // 3.14, -20.0, 1e-1, -10e3
| BOOL // true, TrUe, false, False
| UNQUOTED_CHAR // /, -, \, +, ., $, %, *, @, ?, |
| ESC // \\, \(, \), \[, \], \{, \}, \:, \=, \ , \\t, \,
| WS // whitespaces
)+;

Elements​

Key​

Key is the component before the =. A few examples:

foo.bar           # A config key
hydra/launcher # A config group
group@pkg # A config group assigned to the package pkg
group@pkg1:pkg2 # A config group changing the package from pkg1 to pkg2

Quoted values​

Hydra supports both double quotes and single quoted values. Quoted strings can accept any value between the quotes, but some characters need escaping:

  • to include a single quote in a single quoted string, use \' (for double quotes in a double quoted string, use \")
  • any sequence of \ characters preceding a quote (either an escaped quote as described in the previous point, or the closing quote) must be escaped by doubling the number of \
Double quotes
"hello there"
"escaped \"double quote\""
"the path is C:\\\"some folder\"\\"
"1,2,3"
"{a:10} ${xyz}"
"'single quoted string'"
Single quotes
'hello there'
'escaped \'single quote\''
'the path is C:\\\'some folder\'\\'
'1,2,3'
'{a:10} ${xyz}'
'"double quoted string"'

It may be necessary to use multiple pairs of quotes to prevent your shell from consuming quotation marks before they are passed to hydra.

$ python my_app.py '+foo="{a: 10}"'
foo: '{a: 10}'

$ python my_app.py '+foo={a: 10}'
foo:
a: 10

Here are some best practices around quoting in CLI overrides:

  • Quote the whole key=value pair with single quotes, as in the first two examples above. These quotes are for the benefit of the shell.
  • Do not quote keys.
  • Only quote values if they contain a space. It will work if you always quote values, but it will turn numbers/dicts/lists into strings (as in the first example above).
  • When you are quoting values, use double quotes to avoid collision with the outer single quoted consumed by the shell.

Whitespaces in unquoted values​

Unquoted Override values can contain non leading or trailing whitespaces. For example, msg=hello world is a legal override (key is msg and value is the string hello world). Normally, your shell will interpret values with whitespaces as being multiple parameters (key=a b would be interpreted as key=a and b). To prevent this you can quote them with a single quote. For example:

$ python my_app.py 'msg=hello world'

Note that trailing and leading whitespace are ignored, the above is equivalent to:

$ python my_app.py 'msg=    hello world    '

Escaped characters in unquoted values​

Hydra's parser considers some characters to be illegal in unquoted strings. These otherwise special characters may be included in unquoted values by escaping them with a \. These characters are: \()[]{}:=, \t (the last two ones being the whitespace and tab characters).

As an example, in the following dir is set to the string job{a=1,b=2,c=3}:

$ python my_app.py 'dir=job\{a\=1\,b\=2\,c\=3\}'

As an alternative to escaping special characters with a backslash, the value containing the special character may be quoted:

$ python my_app.py 'dir=A[B'    # parser error
$ python my_app.py 'dir="A[B"' # ok
$ python my_app.py 'dir=A\[B' # ok

Primitives​

  • id : oompa10, loompa_12
  • null: null
  • int: 10, -20, 0, 1_000_000.
  • float: 3.14, -10e6, inf, -inf, nan.
  • bool: true, false
  • dot_path: foo.bar
  • interpolation: ${foo.bar}, ${oc.env:USER,me}

Constants (null, true, false, inf, nan) are case-insensitive.

important

Always single-quote interpolations in the shell, to prevent replacement with shell variables:

$ python my_app.py 'dir=/root/${name}'

In addition, more complex interpolations containing special characters may require being passed within a quoted value (note the extra double quotes surrounding the interpolation):

$ python my_app.py 'dir="${get_dir: {root: /root, name: ${name}}}"'

Dictionaries and Lists​

Lists​

foo=[1,2,3]
nested=[a,[b,[c]]]

Dictionaries​

foo={a:10,b:20}
nested={a:10,b:{c:30,d:40}}

Dictionaries are merged, not assigned. The following example illustrates the point:

Input config
db:
driver: mysql
user: ???
pass: ???
db={user:root,pass:1234}
db:
driver: mysql
user: root
pass: 1234
important

Always single-quote overrides that contains dicts and lists in the shell.

Sweeper syntax​

A choice sweep is comma separated list with two or more elements:

key=a,b                       # Simple sweep: ChoiceSweep(a, b)
key="a,b","c,d" # Elements can be quoted strings, ChoiceSweep("a,b", "c,d")
key=[a,b],[c,d] # Elements can be real lists, ChoiceSweep([a,b], [c,d])
key={a:10, b:20},{c:30,d:40} # And dictionaries: ChoiceSweep({a:10, b:20}, {c:30,d:40})

More sweeping options are described in the Extended Grammar page.

important

You may need to quote your choice sweep in the shell.

Functions​

Hydra supports several functions in the command line. See the Extended Grammar page for more information.

Working with your shell​

All shells interprets command line inputs and may change what is passed to the process. A good way to determine what the shell is doing to your command is to echo it.

# bash output
$ echo foo_{a:10,b:20} ${HOME} [b,c]*
foo_a:10 foo_b:20 /home/omry build_helpers
$ echo 'foo_{a:10,b:20}' '${HOME}' '[b,c]*'
foo_{a:10,b:20} ${HOME} [b,c]*

If in doubt, quote a command line element with a single quote (').

If you want to pass quotes to Hydra in a shell quoted string, it's best to pass double quotes.

$ echo '"hello world"'
"hello world"

You can use some shell specific commands to change their behavior, but the cost will be that their behavior will change.

Bash​

You can disable braces expansion, filename generation (globing) and hist expansion. Please note that this will change your shell behavior for the current session.

$ set +o braceexpand -o noglob +o histexpand
$ echo key1={a:10,b:20} key2=${HOME} key=[b]*
key1={a:10,b:20} key2=/home/omry key=[b]*
# does not help with () though:
$ echo key=choice(a,b,c)
bash: syntax error near unexpected token '('
$ echo 'key=choice(a,b,c)'
key=choice(a,b,c)

Other shells​

Send a PR to add information about your favorite shell here.