[Koha-bugs] [Bug 22417] Delegate background jobs execution

[bugzilla-daemon at bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

Jonathan Druart <jonathan.druart at bugs.koha-community.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |In Discussion

--- Comment #1 from Jonathan Druart <jonathan.druart at bugs.koha-community.org> ---
I first tried Minion but found problematic issues for us (Koha):
- The mysql backend requide Mojo::mysql which requires DBD::mysql 4.042
However Jessie has 4.028 and Stretch 4.041
- We could have used the Postgres backend, but we would have require
MySQL/MariaDB and Postgres, which is not acceptable.

I then gave a try to RabbitMQ (https://www.rabbitmq.com/documentation.html),
which is an implementation of AMQP in Erlang.
If you want more information about it I let you follow the link.
The result is quite good, I managed to have a working POC, adapt to our uses,
in a single day.

Test plan:
0. Prerequisite:
% sudo apt install libanyevent-rabbitmq-perl rabbitmq-server
% sudo cpanm Net::RabbitFoot # Quid of the debian package?

Then make sure the RabbitMQ server is running:
% sudo service rabbitmq-server status
and start it if needed with
% sudo service rabbitmq-server start

Then create, or make sure you have a MARC Modification Template, with
template_id=1 (like add new field 123$z=foo)

1. To test this patch set you should first understand how the enqueue/consume
process work:
 % perl new_koha_job.pl # Launch it 1+ times
 => The job must have been enqueued (no error in the output)
 % perl koha_worker.pl # Will consume the 1+ tasks you enqueued
 Do not close this worker and start another one in another console, then
enqueue more new jobs.

Enjoy :)
(yes it should be fun)

2. Watch the history of the jobs
Hit admin/background_jobs.pl (logged in as superlibrarian)
This is an interface (to improve) to the list of pending/running and finished
processes

3. Use this whole stuff in a real Koha world:
Play with the "Batch record modification" tool (with biblios)
Notice the different in the last (report) step and click on the "View detail of
the enqueued job" link

Discussions:
1. Please grep TODO and FIXME in the patchset, there are lot to do/fix.
2. Even if we do not go for RabbitMQ I think we could start moving the code
from pl to Koha::BackgroundJob::*pm
It will be "easy" (nothing is easy, right?) to replace it with something else.
3. With this base we will want to answer different needs:
  * background jobs
  * task scheduler
  * reindex process (?)

Going further:
1. Implement Koha::BackgroundJob for the different background jobs
2. Provide a koha-worker script to consume to jobs
3. Rewrite the task scheduler with Koha::BackgroundJob (1993)
4. Provide a configuration file to prioritize or postpone jobs (for instance no
batch modification on Wednesday between 9-17) - Or maybe we won't need that (?)
5. What would be your needs?

Code is at https://gitlab.com/joubu/Koha/commits/bug_22417 (as the code will
evolve in the next days I will not pollute here).
Switched to 'In Discussion' to collect thoughts.

-- 
You are receiving this mail because:
You are watching all bug changes.