RQ Launcher plugin

  

The RQ Launcher plugin provides a launcher for distributed execution and job queuing based on Redis Queue (RQ).

RQ launcher allows parallelizing across multiple nodes and scheduling jobs in queues. Usage of this plugin requires a Redis server. When parallelisation on a single node is intended, the Joblib launcher may be preferable, since it works without a database.

Installation

pip install hydra-rq-launcher --upgrade

Usage of this plugin requires a Redis server.

Note that RQ does not support Windows.

Usage

Once installed, add hydra/launcher=rq to your command line. Alternatively, override hydra/launcher in your config:

defaults:
  - override hydra/launcher: rq

The default configuration is as follows:

$ python your_app.py hydra/launcher=rq --cfg hydra -p hydra.launcher

# @package hydra.launcher
_target_: hydra_plugins.hydra_rq_launcher.rq_launcher.RQLauncher
enqueue:
  job_timeout: null
  ttl: null
  result_ttl: null
  failure_ttl: null
  at_front: false
  job_id: null
  description: null
queue: default 
redis:
  host: ${oc.env:REDIS_HOST,localhost}
  port: ${oc.env:REDIS_PORT,6379}
  db: ${oc.env:REDIS_DB,0}
  password: ${oc.env:REDIS_PASSWORD,null}
  ssl: ${oc.env:REDIS_SSL,False}
  ssl_ca_certs: ${oc.env:REDIS_SSL_CA_CERTS,null
  mock: ${oc.env:REDIS_MOCK,False}
stop_after_enqueue: false
wait_polling: 1.0

Further descriptions on the variables can be found in the plugin config file, defined here. There are several standard approaches for configuring plugins. Check this page for more information.

The plugin is using environment variables to store Redis connection information. The environment variables REDIS_HOST, REDIS_PORT, REDIS_DB, and REDIS_PASSWORD, are used for the host address, port, database, and password of the server, respectively. Support for Redis SSL connections is controlled through REDIS_SSL and REDIS_SSL_CA_CERTS; see redis-py documentation.

For example, they might be set as follows when using bash or zsh as a shell:

export REDIS_HOST="localhost"
export REDIS_PORT="6379"
export REDIS_DB="0"
export REDIS_PASSWORD=""

Enable SSL by setting the relevant environment variables. e.g:

export REDIS_SSL=true
export REDIS_SSL_CA_CERTS=/etc/ssl/certs/ca-certificates.crt

Assuming configured environment variables, workers connecting to the Redis server can be launched using:

rq worker --url redis://:$REDIS_PASSWORD@$REDIS_HOST:$REDIS_PORT/$REDIS_DB

An example application using this launcher is provided in the plugin repository.

Starting the app with python my_app.py --multirun task=1,2,3,4,5 will enqueue five jobs to be processed by worker instances:

$ python my_app.py --multirun task=1,2,3,4,5

[HYDRA] RQ Launcher is enqueuing 5 job(s) in queue : default
[HYDRA] Sweep output dir : multirun/2020-06-15/18-00-00
[HYDRA] Enqueued 13b3da4e-03f7-4d16-9ca8-cfb3c48afeae
[HYDRA]     #1 : task=1
[HYDRA] Enqueued 00c6a32d-e5a4-432c-a0f3-b9d4ef0dd585
[HYDRA]     #2 : task=2
[HYDRA] Enqueued 63b90f27-0711-4c95-8f63-70164fd850df
[HYDRA]     #3 : task=3
[HYDRA] Enqueued b1d49825-8b28-4516-90ca-8106477e1eb1
[HYDRA]     #4 : task=4
[HYDRA] Enqueued ed96bdaa-087d-4c7f-9ecb-56daf948d5e2
[HYDRA]     #5 : task=5
[HYDRA] Finished enqueuing
[HYDRA] Polling job statuses every 1.0 sec

Note that any dependencies need to be installed in the Python environment used to run the RQ worker. For serialization of jobs cloudpickle is used.

The RQ documentation holds further information on job monitoring, which can be done via console or web interfaces, and provides patterns for worker and exception handling.