[Koha-bugs] [Bug 22417] Add a task queue

[bugzilla-daemon at bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

--- Comment #326 from David Cook <dcook at prosentient.com.au> ---
(In reply to Marcel de Rooy from comment #325)
> Keep in mind that some libraries might not have the luxury of sufficient
> funds to allocate the resources that most of us do have.
> 

That is true.

I would argue that Elasticsearch requires resources, but we still have Zebra
for less well resourced libraries.

Technically speaking, Koha doesn't require RabbitMQ. Rather, it just requires a
message queue that can use the STOMP protocol. So we could always recommend
RabbitMQ for well resourced libraries, but provide/recommend a smaller
alternative. 

A quick Google suggests ocamlmq as a lightweight option. Apache ActiveMQ is
embedded in DSpace and Fedora Commons I believe, but we could use a standalone
instance. 

Here's a list of STOMP compliant servers:
https://stomp.github.io/implementations.html

> If the load is just the result of the waiting/polling, the thing is really
> badly programmed ;)

I think that's a bit presumptuous. In the Koha community, I think that we
should endeavour to be open-minded about technology.

After chatting with some RabbitMQ folk and briefly studying Erlang, it appears
that the Erlang BEAM virtual machine uses "busy waiting" to stay in a state of
readiness, so that it's able to start new work as quickly as possible. This
helps RabbitMQ to achieve high throughput on heavy workloads. So I wouldn't say
that it is "really badly programmed". Rather, it's a deliberate programming
choice designed for high performance computing (HPC). I think that we should be
careful not to mistake our own local perspectives for absolute truth. 

That being said, I'm not the most trusting person, and don't like to accept
what people tell me without some evidence. While using "+sbwt none" does reduce
the CPU usage and reduce the number of syscalls, it appears to me that when
using RabbitMQ 3.6.10-1ubuntu0.1 with the Erlang Run-Time System (ERTS) 9.2 on
Ubuntu 18.04, the Erlang schedulers are still "busy waiting", despite what the
RabbitMQ and Erlang documentation say. I'm able to demonstrate that using "top"
and "strace".

Someone on the RabbitMQ listserv pointed out that I was using outdated tech,
and https://www.rabbitmq.com/install-debian.html does point out that the
software in the Debian and Ubuntu repositories is often quite out-of-date (just
like Zebra) and suggests installing from their own sources. 

I'd be interested to see if newer versions of RabbitMQ and Erlang do as much
busy waiting. I might do some quick Docker-based experiments to see if anything
obvious jumps out.

-- 
You are receiving this mail because:
You are watching all bug changes.