The RQ Launcher plugin provides a launcher for distributed execution and job queuing based on Redis Queue (RQ).
RQ launcher allows parallelizing across multiple nodes and scheduling jobs in queues. Usage of this plugin requires a Redis server. When parallelisation on a single node is intended, the Joblib launcher may be preferable, since it works without a database.
Usage of this plugin requires a Redis server.
Note that RQ does not support Windows.
Once installed, add
hydra/launcher=rq to your command line. Alternatively, override
hydra/launcher in your config:
The configuration packaged with the plugin is defined here. The default configuration is as follows:
The plugin is using environment variables to store Redis connection information. The environment variables
REDIS_PASSWORD, are used for the host address, port, database, and password of the server, respectively.
For example, they might be set as follows when using
zsh as a shell:
Assuming configured environment variables, workers connecting to the Redis server can be launched using:
An example application using this launcher is provided in the plugin repository.
Starting the app with
python my_app.py --multirun task=1,2,3,4,5 will enqueue five jobs to be processed by worker instances:
Note that any dependencies need to be installed in the Python environment used to run the RQ worker. For serialization of jobs
cloudpickle is used.