[Koha-bugs] [Bug 22417] Delegate background jobs execution

[bugzilla-daemon at bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

--- Comment #47 from Tomás Cohen Arazi <tomascohen at gmail.com> ---
I'm impressed by the results, Jonathan. I really sorry I didn't spend time on
this before. It deserved it for sure.

I will probably repeat myself, but in my opinion what we need is:

- A task_queue table
- A daemon that looks for things to do, and spawns children processes to do the
tasks, mark them as completed or failed, handle retries, etc

That's what basically Minion does.

This implementation:
- Sets a message broker
- Has a table to keep track of tasks
- Sends a message for each task that is enqueued
- Has a daemon that looks for those messages and runs the task

What we are doing is modelling a task queue with a message queue, from which we
use (probably) its scheduling features, and the 'retry' logic perhaps?

I'm not in a situation in which I can provide an alternative right now, so I
wouldn't want to block this right now. So will try to add my constructive two
cents:

- I would like to know what ideas you might have for maintaining the 'tasks'
catalog, specially when it comes to plugins (i.e. the mapping between
background_job.type and the method that has to be run.
- I expected to see an event-driven implementation of the koha_worker.pl loop,
as messages will arrive and we should react to them instead of polling. Maybe
it is a limitation from Net::STOMP, I saw there's AnyEvent::STOMP which is
event-driven. Maybe the 'blocking and waiting for a frame' behaviour from
Net::STOMP::receive_frame works similarly... worth checking.
- I would like to see OO in Koha::BackgroundJob but this is minor abviously

I will organise my work so I can spend some hours on redoing this using
AnyEvent::Task::Server and AnyEvent::Task::Client as I wrote a POC a lot of
time ago.

GOOD JOB Jonathan, it is obvious you spent lots of time on this and you put the
bar very high.

-- 
You are receiving this mail because:
You are watching all bug changes.