[Koha-bugs] [Bug 22417] Add a task queue

[bugzilla-daemon at bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

--- Comment #263 from David Cook <dcook at prosentient.com.au> ---
(In reply to Julian Maurice from comment #260)
> I'm curious about how we would implement a task scheduler with a message
> queue. Would that be with something like
> https://github.com/rabbitmq/rabbitmq-delayed-message-exchange ?
> 

Jonathan has previously suggested that plugin
(https://www.rabbitmq.com/blog/2015/04/16/scheduling-messages-with-rabbitmq/).
It could work.

Task schedulers are a passion of mine, and there are also other ways of doing
them with a message queue. Typically, you'll see to-purpose software like
Celery (https://docs.celeryproject.org/en/stable/index.html) or BigBen
(https://github.com/walmartlabs/bigben) that provide task schedulers that use
message queues.

I've also studied the source code of Perl libraries like AnyEvent, POE,
Mojo::IOLoop, etc as well as the native implementation of timers in the Golang
programming language. Ultimately, task schedulers are all about running a
process (or a thread) that sleeps until a task is scheduled to run. It then
triggers some kind of event. In Golang or Perl, that's whatever function you
defined. In Celery or BigBen, it will be to send a message through the message
queue to worker processes. 

I've written my own task schedulers in Perl and in Golang, and I'm happy to
provide one for Koha later. It's not required right now, but I know what to do
to implement one.

(In reply to Julian Maurice from comment #260)
> The ability to have multiple workers is really great, but how will you run
> them on another server ? You would need to have the whole Koha code
> duplicated on this other server, right ?
> 

Not necessarily. 

If we write code well, we could write workers that could run on standalone
servers/containers. We just need to get out of the monolith mindset. This would
possibly involve building up the HTTP API further, which is a goal we already
have anyway. 

For Koha packages, we could also package that worker code into distinct
packages, which makes it easy to install on other servers/containers.

I plan to do more work on this front in the future for Koha.

(In reply to Julian Maurice from comment #260)
> Do you mean our own message broker ? Because I certainly do not wish that.
> All I'm suggesting is to make the dependency to RabbitMQ (or any other MQ)
> optional, for small installations that won't benefit from it.

Except even small installations *will* benefit from using a message queue.

Right now, Koha - even when running Plack - depends on CGI. Using RabbitMQ
provides a standard way of breaking our remaining dependency on CGI scripts for
long-running tasks. 

As time passes, more and more of Koha will use the message queue as well. There
are many long-running jobs, which would benefit from being placed into the
background. 

This change is about building capacity and improving performance. 

Koha is about being all things to all people, and really all people need
asynchronous processing, especially as users demand smoother user experiences.

-- 
You are receiving this mail because:
You are watching all bug changes.