On Thu, Dec 13, 2018 at 09:29:57AM +0000, Milosz Wasilewski wrote:

Hi,

We had an issue that tasks from one backend blocked tasks from other backends. Maybe we can create as many queues as we have backends and route the tasks to them? I'm not sure this is easy or even possible, but just sending this idea out for comments.

Automatically having one queue per backend is not so easy to do. The problem is that the queues must be defined at the settings (source code) level, while the backends are database objects. There is no easy way to access the db at the point where the settings are being evaluated since the db connection itself is defined as part of the settings.

Now, there is another option. We can:

- define a set of queues in the settings - add a field to the Backend objects where we inform which queue should be used for tasks from that backend. - then use a custom router¹ to choose the queue at runtime

¹ as in https://www.distributedpython.com/2018/06/05/dynamic-task-routing/

IMO this is not hard, but it's also not trivial.