[Koha-bugs] [Bug 32573] background_jobs_worker.pl should ACK a message before it forks and runs the job

[bugzilla-daemon at bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573

--- Comment #8 from David Cook <dcook at prosentient.com.au> ---
(In reply to Kyle M Hall from comment #5)
> Considering batch record modifications are not atomic, I don't think that's
> necessarily a compelling argument.

I think that shows that batch modifications are problematically designed.
Consider my comment on bug 32278 that talks about how having item/record
numbers saved into tracked batches. We could then process those batches in an
atomic fashion. 

> I can't see any scenario where either:
> A) We should ack before processing. We are stymying Koha's ability to do
> background processing by running extrememly long processes in serial with no
> way to fork multiple background jobs at one.
> 
> B) RabbitMQ/STOMP is the wrong tool for the job and we should remove it's
> use from Koha.
>
> Actually, I can revise A a bit. We can fork and give the existing rabbit
> connection to the child process, and have the parent start a new connection
> to pass off again. That mains we can parallel process without acking first.
> It doesn't solve the 30 minutes time limit ( which is why I'm going to
> continue advocating for acking first ) but it does improve the situation
> greatly.

>From day one, I've advocated for multiple worker processes. Frequently, you
start multiple worker processes independently of the web app. I have other
applications where I run the worker processes in standalone containers and I
can start as many as I want to do the work based on the workload of that
application (based on the computational power I have available of course). I
also put them together in pipelines which means I can process lots of data very
quickly. Those apps are way more data heavy than Koha. I still ACK after I
process my RabbitMQ messages. 

In Koha, we shouldn't be having extremely long running processes. We don't do
any 1 thing that is really computationally that heavy; we're not running
complex image manipulation on high resolution medical images with terabytes
worth of data. We're just writing inefficient suboptimal code. 

RabbitMQ isn't the wrong tool for the job, but I think the Koha community
doesn't have the knowledge/experience to know how to use it (well). And I'll
include myself in that camp. I think that I've probably used RabbitMQ more than
anyone else in the Koha community, and I'd still consider myself a RabbitMQ
newbie.

-- 
You are receiving this mail because:
You are watching all bug changes.