[Koha-bugs] [Bug 22417] Delegate background jobs execution

[bugzilla-daemon at bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

--- Comment #48 from David Cook <dcook at prosentient.com.au> ---
(In reply to Tomás Cohen Arazi from comment #47)
> I'm impressed by the results, Jonathan. I really sorry I didn't spend time
> on this before. It deserved it for sure.
>

Ditto. I would love to see more interest in this one, as it could be such a
valuable addition to Koha. I don't know why more people don't seem interested.
Not sure if they're just uninterested because they're not familiar with the
technologies.

> I will probably repeat myself, but in my opinion what we need is:
> 
> - A task_queue table
> - A daemon that looks for things to do, and spawns children processes to do
> the tasks, mark them as completed or failed, handle retries, etc
>

This is similar to the model that I've used in the past (for Bug 10662 and some
of my fun projects), and it has pros and cons. Personally, I am interested in a
model where we create a task scheduler daemon that Koha connects to via a HTTP
API, and the task scheduler daemon publishes to the message broker when it's
time to run a task. It has pros and cons too:

FORKING:
- One pro is that it closely couples the task scheduler to Koha, which makes it
very easy to *do things*. 
- One con is that it can only use 1 machine's resources, which reduces its
availability and scalability. It also means you're probably running 1 task
scheduler per Koha instance, which could increase management/maintenance
complexity.
- One pro is that it's very easy to increase/decrease limits on forking child
processes.
- One pro is that it would be easier to use plugins to do background jobs, as
the task scheduler would just need to load the code.

WORKERS:
- One pro is that it loosely coupled the task scheduler to Koha, which means it
is a discrete modular unit of code that can be developed and tested
independently of the rest of the Koha codebase. 
- One con is that being loosely coupled makes it harder to do things. It means
using more APIs instead of giving direct database access and internal (ie Perl)
API access to workers. 
- One pro is that a sysadmin has a lot of control for increasing/decreasing the
number of worker processes, and where they run. (This lends itself to
containerization.)
- One pro is that it means workers can be written in any programming language.
Tasks which require high efficiency could have workers written in a more high
performing language. 
- One con is that it increases complexity. Managing worker processes is harder
than forking new child processes. Supervisord could be useful for this though.
This is a solved problem.
- One con is that plugins wouldn't be able to introduce worker processes.
(Slowly, in my free time, I'm developing a Koha plugin which will connect to a
Golang-based task scheduler using a HTTP API which runs OAI-PMH harvest jobs
using goroutines. The downside is that a sysadmin would have to manage the
Golang task scheduler. It wouldn't work just out of the box after loading the
Koha plugin.)

-- 
You are receiving this mail because:
You are watching all bug changes.