[Koha-bugs] [Bug 22417] Delegate background jobs execution

[bugzilla-daemon at bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

--- Comment #38 from David Cook <dcook at prosentient.com.au> ---
(In reply to Julian Maurice from comment #35)
> STOMP is very simple indeed. But combined with RabbitMQ it becomes a
> powerful solution for dispatching messages. And that's not a bad thing of
> course, but I think it adds a useless layer of complexity in our case.

I am willing to try both Minion and RabbitMQ (and I keep telling myself I will
try them soon in my personal time since I'm staying home due to the pandemic).

But I want to say first that I don't think we'd be adding a useless layer of
complexity with RabbitMQ. Provided the implementation is well-architected, I
think message queues are an integral part of scaling up an application and
improving interoperability. 

Using a language agnostic protocol like STOMP, it could be easier to implement
higher performance workers in other languages. For instance, maybe we'd want to
implement some workers in Golang, which makes concurrent programming very easy
with its implementation of user threads as goroutines balanced against kernel
threads depending on resource demands. 

For my work on a OAI-PMH harvester, I have been thinking of rewriting it in
something like Golang instead of Perl, and then hooking into Koha using a
standard message queue (instead of my bespoke POE-based implementation) and/or
APIs. (Honestly though my Perl version is quite high performance in terms of
downloads but the bottleneck is importing into Koha. That's still slower than
I'd like. I occasionally think about how to optimally implement a scalable high
performance MARCXML record importer. But that's a whole other topic really.)

That said, RabbitMQ is a new dependency, and I think our collective fear of new
dependencies has kept us from implementing a message queue previously. 

I think we've arrived at a point where we *have* to have some kind of
job/message queue though. 

Anyway, sorry for the ramble. It's just a topic in which I have a strong
interest. I'm going to put it at the top of my personal to try pile.

-- 
You are receiving this mail because:
You are watching all bug changes.