The Joblib Launcher plugin provides a launcher for parallel tasks based on
Once installed, add
hydra/launcher=joblib to your command line. Alternatively, override
hydra/launcher in your config:
By default, process-based parallelism using all available CPU cores is used. By overriding the default configuration, it is e.g. possible limit the number of parallel executions.
The JobLibLauncherConf backing the config is defined here:
You can discover the Joblib Launcher parameters with:
There are several standard approaches for configuring plugins. Check this page for more information.
Joblib.Parallel documentation for full details about the parameters above.
An example application using this launcher is provided in the plugin repository.
Starting the app with
python my_app.py --multirun task=1,2,3,4,5 will launch five parallel executions: