[Koha-bugs] [Bug 22417] Delegate background jobs execution

[bugzilla-daemon at bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

Jonathan Druart <jonathan.druart at bugs.koha-community.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |Needs Signoff

--- Comment #30 from Jonathan Druart <jonathan.druart at bugs.koha-community.org> ---
So, getting back to this (one year after the previous try).

Please read this comment, the different commit, then try the patchset.

I have changed some stuffs, in order to make everybody (hopefully) happy.
I removed the dependency to Net::RabbitFoot, in order to not depend on
non-packaged stuffs, and I decided to take a slightly different direction.
This patch can be considered ready for testing.

I have investigated a bit what could be possible, and picked STOMP [1]:
"""
STOMP is the Simple (or Streaming) Text Orientated Messaging Protocol.

STOMP provides an interoperable wire format so that STOMP clients can
communicate with any STOMP message broker to provide easy and widespread
messaging interoperability among many languages, platforms and brokers.
"""

There is an actively developed Perl library, Net::Stomp, packaged for debian.

I think it could be a very good first step, as we could have a dependency on a
small server at first, and sysop could then use more powerful/scalable/etc
server depending on their needs.
However I sticked to RabbitMQ for the server in my tests, and it worked great.

Something else worth to be noted: STOMP does not provide a 'namespace'
parameter that can be specified at connection time, unlike Net::RabbitFoot. But
there is a 'destination' on send/receive (actually when subscribing), that
seems to do the same job.

There is a new patch set (see the same remote branch, previous one is pushed to
old/20200225_bug_22417), with different follow-ups adding and fixing several
things (better error handling , fix batch authority modification, etc.)

Test plan:
0. Prerequisite:
% apt install rabbitmq-server # server
% rabbitmq-plugins enable rabbitmq_stomp # Enagle the STOMP plugin
% sudo service rabbitmq-server status # Make sure the RabbitMQ server is
running
% sudo service rabbitmq-server start # start it if needed

% apt install libnet-stomp-perl # install the client lib
% cp debian/scripts/koha-functions.sh /usr/share/koha/bin/koha-functions.sh
% cp debian/scripts/koha-worker /usr/bin/
% koha-worker --start kohadev # start the background jobs worker
% tail -f /var/log/rabbitmq/* # take a look at the look to understand what's
going on
% Execute the updatedatabase entry to create the new DB table

Then create, or make sure you have a MARC Modification Template, with
template_id=1 (like add new field 123$z=foo)

1. To test this patch set you should first understand how the enqueue/consume
process work:
 % perl new_koha_job.pl # Launch it 1+ times
 => The job must have been enqueued (no error in the output)
 % perl koha_worker.pl # Will consume the 1+ tasks you enqueued
 Do not close this worker and start another one in another console, then
enqueue more new jobs.

Use the following bash loop to enqueue lot of jobs:
  % for i in {1..100}; do perl new_koha_job.pl; done

Enjoy :)
(yes it should be fun)

2. Watch the history of the jobs
Hit admin/background_jobs.pl (logged in as superlibrarian)
This is an interface of the list of pending/running and finished processes

3. Use this whole stuff in a real Koha world:
Play with the "Batch record modification" tool (with biblios or authorities)
Notice the different in the last (report) step and click on the "View detail of
the enqueued job" link

Discussions:
1. Please grep TODO and FIXME in the patchset, there are lot to do/fix.
2. Even if we do not go for STOMP I think we could start moving the code from
pl to Koha::BackgroundJob::*pm
It will be "easy" (nothing is easy, right?) to replace it with something else.
3. With this base we will want to answer different needs:
  * background jobs
  * task scheduler
  * reindex process (?)

Going further:
1. Implement Koha::BackgroundJob for the different background jobs
2. Rewrite the task scheduler with Koha::BackgroundJob (1993)
3. Provide a configuration file to prioritize or postpone jobs (for instance no
batch modification on Wednesday between 9-17) - Or maybe we won't need that (?)
4. Server-side, there is the possibility to configure a lot of things, like
users, destinations, ssl, etc. By default guest/guest has a localhost access
only.

Code is at https://gitlab.com/joubu/Koha/commits/bug_22417

[1] https://stomp.github.io/
    https://stomp.github.io/stomp-specification-1.2.html

-- 
You are receiving this mail because:
You are watching all bug changes.