[Koha-bugs] [Bug 22417] Add a task queue

[bugzilla-daemon at bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

--- Comment #161 from David Cook <dcook at prosentient.com.au> ---
(In reply to Marcel de Rooy from comment #160)
> No, that is not my suggestion here. But this argument is about its
> architecture. We use a message queue for a more specific goal. This has
> translated into the actual code making it harder to widen the scope again.
> Later on, I want to add messages (no tasks) in another area but what if I
> dont want to go via BackgroundJob?

I had the same thought. 

Locally, I'm already using RabbitMQ for some experimental features, and I have
a Koha::MQ class that takes care of connections, configuration, etc.

I use Koha::MQ in my Web UI to send a fire and forget message to a background
service. It's a very light implementation.

> The last few sentences actually ask for a justification why send we the same
> data to both a table and the message queue. What is in this specific case
> the added value of the message queue? If we only use it for this specific
> case, it looks like much overhead for what it brings. If the justification
> is in the wider use of a message queue, lets translate that to its design.

You need the database table to track the task. The database table allows you to
see its progress, store results, etc. Otherwise, you're just doing a fire and
forget (like I do above), which is problematic, as there's no way of tracking
the task to see if it succeeded or what its output was. 

But the message queue is more efficient and scalable than polling the database.
With the message queue, workers use a select(2) syscall to wait until they have
data to read on the socket connection they have to the message broker. They
don't consume CPU time. They just sleep. They're only woken up when they have a
message from the message broker. When polling the database, you're generating
steady network and file I/O, so then you tune the polling time to something
that doesn't impact your system too much, which takes time/energy/experience
and is ultimately slower than the message queue. 

Plus, if you were polling the database, you'd only be able to have 1 process
doing that. You couldn't have 5 processes polling the database. So then your
process polling the database becomes a de facto message broker anyway, and then
you're re-inventing the wheel (and spokes).

-- 
You are receiving this mail because:
You are watching all bug changes.