https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Fridolin SOMERS <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #2 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- And the screenshots: https://snag.gy/WVeqDf.jpg What you get once the job has been enqueued https://snag.gy/1GgnN5.jpg Detail of a non started job https://snag.gy/ySnlaF.jpg Detail of a finished job https://snag.gy/s5dF7n.jpg List of jobs -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Bob Birchall <bob@calyx.net.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bob@calyx.net.au -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #4 from David Cook <dcook@prosentient.com.au> --- (In reply to Jonathan Druart from comment #1)

Discussions: <snip> 2. Even if we do not go for RabbitMQ I think we could start moving the code from pl to Koha::BackgroundJob::*pm It will be "easy" (nothing is easy, right?) to replace it with something else.

I totally agree; it would nice to be as message queue implementation agnostic as possible.

Going further: 1. Implement Koha::BackgroundJob for the different background jobs 2. Provide a koha-worker script to consume to jobs 3. Rewrite the task scheduler with Koha::BackgroundJob (1993)

It's been a couple years since I looked at RabbitMQ. Is there a way to cancel/remove/delete a message once it is enqueued? I think this would be useful for a task scheduler. I built that into the POE-based task scheduler I built for https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662. (Actually, I also built-in the ability for in-progress jobs to be canceled in the event a big batch job was started and a person realized it was a mistake.)

4. Provide a configuration file to prioritize or postpone jobs (for instance no batch modification on Wednesday between 9-17) - Or maybe we won't need that (?)

Fridolin mentioned something like this to me when I was talking about https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 with him at Kohacon18 in Portland. He pointed out that for a multi-tenant server you might not want all the different Koha instances on the server to run their background jobs at the same time due to restraints like performance, rate-limiting, connection limits, etc. Off the top of my head, I don't have a solution for this one though, although in the case of https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 I use separate task schedulers for every Koha instance on the server. In this case, we could potentially use 1 RabbitMQ server, and X number of workers... although we'd have to have a way of setting the KOHA_CONF for each task. That might be the best way to handle multi-tenant scenarios. Alternatively, I think we'd need to have unique queues for each Koha instance and workers for those unique queues. Requires a bit of thought in any case.

5. What would be your needs?

Ideally, it would be nice to replace my home-made task scheduler in https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 with this one. I like that mine can add/start/pause/stop/remove tasks, but I haven't seen a mainstream task scheduler that is capable of doing that, so that might be too difficult to do here. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |julian.maurice@biblibre.com -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Josef Moravec <josef.moravec@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |josef.moravec@gmail.com --- Comment #6 from Josef Moravec <josef.moravec@gmail.com> --- I just tried it and it works like a charm! Great job Jonathan! I think just to polish this functionality would be great start and big progress from what we have now. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #8 from David Cook <dcook@prosentient.com.au> --- (In reply to Josef Moravec from comment #6)

I think just to polish this functionality would be great start and big progress from what we have now.

Actually, I think that's a really good point. I think in the community we sometimes have a tendency to debate/discuss things to death... which stifles innovation. I think it could be good to get something polished into Koha (maybe just 1 background job using RabbitMQ to start) and then if that is working well then we add more background jobs. I suppose it might be more efficient to have it fully specified before implementing in Koha rather than doing iterative development but... -- Oh, but I think my concerns about multi-tenant environments are important. At a glance, this implementation seems like it would only allow 1 Koha instance talk to 1 RabbitMQ instance. But I think in practice there are many people who run multiple Koha instances on the same server and it would be good if they could share 1 RabbitMQ instance. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |katrin.fischer@bsz-bw.de --- Comment #10 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Marcel de Rooy from comment #9)

Do we really need the background jobs ? Or should we refactor these scripts in the direction of AJAX API calls in JavaScript ?

I am not sure if this would also allow reimplementing the scheduler with scheduled reports? In my understanding some jobs might not be triggered by the GUI. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #12 from Tomás Cohen Arazi <tomascohen@gmail.com> --- (In reply to Marcel de Rooy from comment #11)

(In reply to Katrin Fischer from comment #10)

...

(In reply to Marcel de Rooy from comment #9)

...

Do we really need the background jobs ? Or should we refactor these scripts in the direction of AJAX API calls in JavaScript ?

I am not sure if this would also allow reimplementing the scheduler with scheduled reports? In my understanding some jobs might not be triggered by the GUI.

OK That's true. I was focused on the forking scripts.

IMHO, in either case we need a way to throttle concurrency and server loads. So a task queue is needed. My only doubt is if we should really model the task queue with a message queue. I tried the same Jonathan tried but using AnyEvent::Task and the results where similar, and it only required a unix socket to communicate client and server processes. My suggestion was to use zeromq. But lets see how this goes. Specially in the integration step. I agree with David that this of course would require a single RabbitMQ server so questions: Is there a concept of namespaces? can any Koha instance access other instance's messages? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #14 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- There is also: "Going further: 2. Provide a koha-worker script to consume the jobs" So one worker per instance. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #16 from David Cook <dcook@prosentient.com.au> --- (In reply to Tomás Cohen Arazi from comment #12)

My only doubt is if we should really model the task queue with a message queue. I tried the same Jonathan tried but using AnyEvent::Task and the results where similar, and it only required a unix socket to communicate client and server processes. My suggestion was to use zeromq. But lets see how this goes. Specially in the integration step.

Yeah, for #10662, I used POE::Component::JobQueue and a Unix socket. It was very effective, although my bespoke implementation had a number of limitations. I figured allowing TCP sockets also might be a good idea just to allow for more distributed architectures. When I was working on #10662, I was considering both RabbitMQ and ZeroMQ, but ended up doing my own job queue. I'm not necessarily sold on RabbitMQ yet, but I figure if Jonathan is willing to do it, I'm happy to see how this goes too. I'm not willing to put in the time to make it happen :/. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #18 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- TODO, remove that: Bug 15032: UpdateDB entry + # $dbh->do( "ALTER TABLE biblio ADD COLUMN badtaste int" ); Bug 22417: Add the ability to cancel a job + sleep(5); :) -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #19 from David Cook <dcook@prosentient.com.au> --- I was thinking about this work this morning. I'm really busy at the moment, but still think this is a good thing to be working on. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #21 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to Julian Maurice from comment #20)

I'm a little concerned about the potential loss of jobs with the RabbitMQ option. Is there any way to avoid this problem ?

Using Monit to make sure the service is always up? :) But indeed we should return an error if the server has not been reached, and so the job not enqueued. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #23 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- But then the job will be in the DB table, so we will not lose it, right? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Juan Romay Sieira <juan.sieira@xercode.es> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |juan.sieira@xercode.es --- Comment #25 from Juan Romay Sieira <juan.sieira@xercode.es> --- Thanks Jonathan for the great job done. I have been testing the development, it works correctly and it seems to me a very good idea to use RabbitMQ as a queue manager. As I had told you by email, we have to do a development focused on the reports, so that they can be executed regardless of AT, and also can be scheduled reports that require user parameters at the time of execution. Because the reports should be able to be programmed, the way to work with RabbitMQ would be through two possibilities: * Use version 3.5.3 or later of RabbitMQ. The one I have installed with Debian Jessie is 3.3.5. From version 3.5.3 you can use a plugin for delayed messages, so that these can be consumed in the future, and not immediately, which is how RabbitMQ works. I do not know how Net::RabbitFoot will behave in this case ... * Use an intermediate table (or the background_jobs table, with a new column called exec_on) for future messages, and have a producer (cronjob) to send the messages to RabbitMQ at the time they need to be consumed. If nobody raises objections to the use of RabbitMQ, I can investigate the above options, and make a POC for scheduled reports. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #27 from David Cook <dcook@prosentient.com.au> --- (In reply to David Cook from comment #26)

Anyway, mostly just playing devil's advocate and trying to think of edge cases.

I suppose the advantage of the cronjob is that system administrators would control the timing of it. I suppose there could be a warning on the page saying that the timing of tasks scheduled for the future would be up to the discretion of the system administrator? Lately, I've been wondering how we can modernize Koha, so that it can run in large distributed environments... but also on low budget single server systems. Being all things to all people... -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Ray Delahunty <r.delahunty@arts.ac.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |r.delahunty@arts.ac.uk -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #28 from Michal Denar <black23@gmail.com> --- Hello, Rabbit is pretty useful for Koha in my eyes. Now we can fight with some limitation because message_que is "one piece" If some message is in the que and waiting for send isn't possible delete just one or group of them. So fragmentation based on RabbitMQ can be great solution. But this bring many new possibillities and better services to borrowers and stuff. I vote for this implementation. May I help with testing or something else? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Discussion |ASSIGNED -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Needs Signoff --- Comment #30 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- So, getting back to this (one year after the previous try). Please read this comment, the different commit, then try the patchset. I have changed some stuffs, in order to make everybody (hopefully) happy. I removed the dependency to Net::RabbitFoot, in order to not depend on non-packaged stuffs, and I decided to take a slightly different direction. This patch can be considered ready for testing. I have investigated a bit what could be possible, and picked STOMP [1]: """ STOMP is the Simple (or Streaming) Text Orientated Messaging Protocol. STOMP provides an interoperable wire format so that STOMP clients can communicate with any STOMP message broker to provide easy and widespread messaging interoperability among many languages, platforms and brokers. """ There is an actively developed Perl library, Net::Stomp, packaged for debian. I think it could be a very good first step, as we could have a dependency on a small server at first, and sysop could then use more powerful/scalable/etc server depending on their needs. However I sticked to RabbitMQ for the server in my tests, and it worked great. Something else worth to be noted: STOMP does not provide a 'namespace' parameter that can be specified at connection time, unlike Net::RabbitFoot. But there is a 'destination' on send/receive (actually when subscribing), that seems to do the same job. There is a new patch set (see the same remote branch, previous one is pushed to old/20200225_bug_22417), with different follow-ups adding and fixing several things (better error handling , fix batch authority modification, etc.) Test plan: 0. Prerequisite: % apt install rabbitmq-server # server % rabbitmq-plugins enable rabbitmq_stomp # Enagle the STOMP plugin % sudo service rabbitmq-server status # Make sure the RabbitMQ server is running % sudo service rabbitmq-server start # start it if needed % apt install libnet-stomp-perl # install the client lib % cp debian/scripts/koha-functions.sh /usr/share/koha/bin/koha-functions.sh % cp debian/scripts/koha-worker /usr/bin/ % koha-worker --start kohadev # start the background jobs worker % tail -f /var/log/rabbitmq/* # take a look at the look to understand what's going on % Execute the updatedatabase entry to create the new DB table Then create, or make sure you have a MARC Modification Template, with template_id=1 (like add new field 123$z=foo) 1. To test this patch set you should first understand how the enqueue/consume process work: % perl new_koha_job.pl # Launch it 1+ times => The job must have been enqueued (no error in the output) % perl koha_worker.pl # Will consume the 1+ tasks you enqueued Do not close this worker and start another one in another console, then enqueue more new jobs. Use the following bash loop to enqueue lot of jobs: % for i in {1..100}; do perl new_koha_job.pl; done Enjoy :) (yes it should be fun) 2. Watch the history of the jobs Hit admin/background_jobs.pl (logged in as superlibrarian) This is an interface of the list of pending/running and finished processes 3. Use this whole stuff in a real Koha world: Play with the "Batch record modification" tool (with biblios or authorities) Notice the different in the last (report) step and click on the "View detail of the enqueued job" link Discussions: 1. Please grep TODO and FIXME in the patchset, there are lot to do/fix. 2. Even if we do not go for STOMP I think we could start moving the code from pl to Koha::BackgroundJob::*pm It will be "easy" (nothing is easy, right?) to replace it with something else. 3. With this base we will want to answer different needs: * background jobs * task scheduler * reindex process (?) Going further: 1. Implement Koha::BackgroundJob for the different background jobs 2. Rewrite the task scheduler with Koha::BackgroundJob (1993) 3. Provide a configuration file to prioritize or postpone jobs (for instance no batch modification on Wednesday between 9-17) - Or maybe we won't need that (?) 4. Server-side, there is the possibility to configure a lot of things, like users, destinations, ssl, etc. By default guest/guest has a localhost access only. Code is at https://gitlab.com/joubu/Koha/commits/bug_22417 [1] https://stomp.github.io/ https://stomp.github.io/stomp-specification-1.2.html -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #32 from Julian Maurice <julian.maurice@biblibre.com> --- Hi, I played again, 1 year later, with this patchset and also with Minion. The STOMP && RabbitMQ solution looks very powerful, but I think that it is a little bit too much for our needs. I made a POC with Minion for comparison. https://gitlab.com/jajm/Koha/-/commits/bug/22417-minion There are 2 commits, there's a test plan in each commit message. About the packaging problems mentioned in comment 1

I first tried Minion but found problematic issues for us (Koha): - The mysql backend requide Mojo::mysql which requires DBD::mysql 4.042 However Jessie has 4.028 and Stretch 4.041 This is still true, but buster is out for a while now and has a more recent version of DBD::mysql, so it should not be a problem to package Mojo::mysql and Minion::Backend::mysql. Older distributions could use the SQLite backend which works well too.

-- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #34 from Julian Maurice <julian.maurice@biblibre.com> --- (In reply to Jonathan Druart from comment #33)

So you need an additional DB. 2 tables to store the jobs (one for our data, to keep track of them, one internal for minion).

What ? No. You can use the same database, and you do not have to keep track of jobs manually. Minion does that for you. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #36 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to Julian Maurice from comment #34)

(In reply to Jonathan Druart from comment #33)

...

So you need an additional DB. 2 tables to store the jobs (one for our data, to keep track of them, one internal for minion).

What ? No. You can use the same database, and you do not have to keep track of jobs manually. Minion does that for you.

Does it allow additional data? Will you manage to write the dashboard I provided with my patches? When I did my different tries bug 15032 it was not possible (not Minion). -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #38 from David Cook <dcook@prosentient.com.au> --- (In reply to Julian Maurice from comment #35)

STOMP is very simple indeed. But combined with RabbitMQ it becomes a powerful solution for dispatching messages. And that's not a bad thing of course, but I think it adds a useless layer of complexity in our case.

I am willing to try both Minion and RabbitMQ (and I keep telling myself I will try them soon in my personal time since I'm staying home due to the pandemic). But I want to say first that I don't think we'd be adding a useless layer of complexity with RabbitMQ. Provided the implementation is well-architected, I think message queues are an integral part of scaling up an application and improving interoperability. Using a language agnostic protocol like STOMP, it could be easier to implement higher performance workers in other languages. For instance, maybe we'd want to implement some workers in Golang, which makes concurrent programming very easy with its implementation of user threads as goroutines balanced against kernel threads depending on resource demands. For my work on a OAI-PMH harvester, I have been thinking of rewriting it in something like Golang instead of Perl, and then hooking into Koha using a standard message queue (instead of my bespoke POE-based implementation) and/or APIs. (Honestly though my Perl version is quite high performance in terms of downloads but the bottleneck is importing into Koha. That's still slower than I'd like. I occasionally think about how to optimally implement a scalable high performance MARCXML record importer. But that's a whole other topic really.) That said, RabbitMQ is a new dependency, and I think our collective fear of new dependencies has kept us from implementing a message queue previously. I think we've arrived at a point where we *have* to have some kind of job/message queue though. Anyway, sorry for the ramble. It's just a topic in which I have a strong interest. I'm going to put it at the top of my personal to try pile. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #40 from David Cook <dcook@prosentient.com.au> --- Ok I'm finally trying this out! (I should be Netflixing or chilling or something, but I am really keen to give this a go, especially after a day of talking about message queues for other projects. Plus I'm always saying we need to modernize, so I really want to help out Jonathan with all his efforts on this one, and show that it's not for nothing.) My steps: 0a. cd ~/git/koha-testing-docker/ 0b. sh run.sh (my own script to for invoking docker-compose with a custom file) 1. winpty docker exec -it koha_koha_1 bash 2. cd koha 3. git remote add joubu https://gitlab.com/joubu/Koha.git 4. git fetch joubu 5. git checkout -b rabbitmq joubu/bug_22417 #The following could probably be handled using https://hub.docker.com/_/rabbitmq 6. apt install rabbitmq-server 7. rabbitmq-plugins enable rabbitmq_stomp 8. service rabbitmq-server start 9. service rabbitmq-server status 10. apt install libnet-stomp-perl 11. cp debian/scripts/koha-functions.sh /usr/share/koha/bin/koha-functions.sh 12. cp debian/scripts/koha-worker /usr/bin/ 13. koha-worker --start kohadev 14. koha-shell kohadev -c installer/data/mysql/updatedatabase.pl 15. restart_all 16. Log into http://localhost:8081/ 17. Go to Tools > MARC modification templates 18. Make new template 18b. Name: test 19. Make new action for "test" template 19b. Add new field "856" "u" "https://koha-community.org" 19c. Description: "RabbitMQ test" 20. koha-shell kohadev 21. perl new_koha_job.pl Output: [x] Enqueuing BatchUpdateBiblio mmtid=1 with biblionumber=1,2,3 22. exit 23. Go to http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 24. Confirm that the job has completed 25. Go to http://localhost:8081/cgi-bin/koha/tools/batch_record_modification.pl 26. Enter list of record numbers: 1, 2, 3, 4, 5 27. Select "test" template 28. Click "Continue" 29. Click "Modify selected records" 30. See message "The job has been enqueued! It will be processed as soon as possible (FIXME - well, it could depend on a config?)" 31. Click "View detail of the enqueued job" 32. Confirm job has status of "finished" 33. Click "Return to the job list" (or "New batch record modification") Great work, Jonathan! The test works beautifully. I'll take a peek at the code now to look for some of those FIXME and TODO messages you mention. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #42 from David Cook <dcook@prosentient.com.au> --- I am tempted to try re-engineering https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 using STOMP and RabbitMQ, as I would really like to use a standard message queue rather than my bespoke implementation. I would also be interested in using plugins to provide a Koha CRUD web interface for managing STOMP messages for specific types of tasks. For instance, I could provide a plugin for sending add/start/stop/update/remove messages to an OAI-PMH harvester. As the sysadmin, I'd need to run up the harvester in the background first, but that would be OK. It would be more so for a proof of concept than anything at first. Anyway definitely bed time now... -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #44 from David Cook <dcook@prosentient.com.au> --- I've also been thinking about the separate needs for a message queue and a task scheduler. For instance, "Bug 1993 - Task Scheduler Needs Re-write" and an OAI-PMH harvester would both need a task scheduler to make sure tasks run at schedule times. However, a Staged MARC Import doesn't need to be scheduled. It just needs to be run ASAP in the background. (Same case for batch modifications, label creation, and many other long-running tasks.) That being said, for scheduled reports (like in "Bug 1993 - Task Scheduler Needs Re-write"), the scheduled task itself might be to enqueue a message into the message queue. (For instance, let's say 20 librarians schedule their SQL reports to run at 6pm. At 6pm, those 20 SQL reports should be put into a queue, and then 1+ report workers can do the work and record the results. They then queue an email job, and emails go out with the results.) (Arguably email processing could be a scheduled task, as you may or may not want emails going out from the library in the middle of the night.) -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version|unspecified |master CC| |kyle@bywatersolutions.com, | |nick@bywatersolutions.com --- Comment #46 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Clap Clap Clap -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #47 from Tomás Cohen Arazi <tomascohen@gmail.com> --- I'm impressed by the results, Jonathan. I really sorry I didn't spend time on this before. It deserved it for sure. I will probably repeat myself, but in my opinion what we need is: - A task_queue table - A daemon that looks for things to do, and spawns children processes to do the tasks, mark them as completed or failed, handle retries, etc That's what basically Minion does. This implementation: - Sets a message broker - Has a table to keep track of tasks - Sends a message for each task that is enqueued - Has a daemon that looks for those messages and runs the task What we are doing is modelling a task queue with a message queue, from which we use (probably) its scheduling features, and the 'retry' logic perhaps? I'm not in a situation in which I can provide an alternative right now, so I wouldn't want to block this right now. So will try to add my constructive two cents: - I would like to know what ideas you might have for maintaining the 'tasks' catalog, specially when it comes to plugins (i.e. the mapping between background_job.type and the method that has to be run. - I expected to see an event-driven implementation of the koha_worker.pl loop, as messages will arrive and we should react to them instead of polling. Maybe it is a limitation from Net::STOMP, I saw there's AnyEvent::STOMP which is event-driven. Maybe the 'blocking and waiting for a frame' behaviour from Net::STOMP::receive_frame works similarly... worth checking. - I would like to see OO in Koha::BackgroundJob but this is minor abviously I will organise my work so I can spend some hours on redoing this using AnyEvent::Task::Server and AnyEvent::Task::Client as I wrote a POC a lot of time ago. GOOD JOB Jonathan, it is obvious you spent lots of time on this and you put the bar very high. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #49 from David Cook <dcook@prosentient.com.au> --- (In reply to Tomás Cohen Arazi from comment #47)

- I would like to know what ideas you might have for maintaining the 'tasks' catalog, specially when it comes to plugins (i.e. the mapping between background_job.type and the method that has to be run.

I am really interested in this too.

- I expected to see an event-driven implementation of the koha_worker.pl loop, as messages will arrive and we should react to them instead of polling. Maybe it is a limitation from Net::STOMP, I saw there's AnyEvent::STOMP which is event-driven. Maybe the 'blocking and waiting for a frame' behaviour from Net::STOMP::receive_frame works similarly... worth checking.

As you say, "blocking and waiting for a frame" is event-driven behaviour, so koha_worker.pl is already event-driven. Net::STOMP and AnyEvent::STOMP both use a select-based event loop. (AnyEvent can use other event loops apparently, but you have to load them. Plus the performance gained using a more sophisticated event loop is likely marginal for our use cases.)

I will organise my work so I can spend some hours on redoing this using AnyEvent::Task::Server and AnyEvent::Task::Client as I wrote a POC a lot of time ago.

That looks interesting! Can I suggest that you use a TCP socket rather than a Unix socket? For my POE-based task scheduler in Bug 10662 (which was similar to this but included timers for task scheduling) I used a Unix socket and it worked fine, but it meant my scheduler has to be on the same machine; in hindsight, I wish I had used a TCP socket. With a TCP socket, it would make Dockerization easier and could make production deployments easier (depending on how the workers are done). Actually, how do you see AnyEvent::Task fitting in with RabbitMQ or task scheduling? The AnyEvent::Task::Server would sit in the background... would your koha_worker.pl be the RabbitMQ consumer and AnyEvent::Task::Client? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |rel_20_11_target, | |RM_priority -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #52 from David Cook <dcook@prosentient.com.au> --- Oh and in the JSON message I specify "class" and "method" (much like the Koha::Plugin::Handler). It also provides "args". So it loads the "class", and runs the "method" with the "args". It's pretty simple. If I were to record results, I'd probably have a "result_callback_method" or something too. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #105542|1 |0 is patch| | Attachment #105542|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #55 from David Cook <dcook@prosentient.com.au> --- At Tomas's request, I've attached a patch I've done for a local customization to use RabbitMQ. It's very simple. I don't have any time-based "task scheduling" per se. It's just FIFO. The mq_scheduler.pl script is problematic in a number of ways, but it's just an experiment. Early testing is good. You submit a form on the Koha web UI, it sends a message to RabbitMQ, the mq_scheduler.pl receives the message from RabbitMQ, the mq_scheduler.pl loads a module and runs some code (which in this case also sends an email), and then it marks the message complete/acknowledged in RabbitMQ. This patch does require a person to run "add_vhost" and "set_permissions" manually using rabbitmqctl on the host running RabbitMQ. However, as I'm defining the queues in koha-conf.xml, the root vhost could be used, and the queues could be prefixed with __DB_NAME__. At this point, I'm fairly indifferent to the method. I like vhosts for the logical separation of the messages, but if it's not practical for a package installation, then maybe we should just use the prefixed queues. The systemd service is *very* package installation friendly. We'd need to update the debian/* files accordingly, but it would be so easy. I haven't done that yet, as I'm testing on non-package installations first. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Delegate background jobs |Add a task queue |execution | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #57 from David Cook <dcook@prosentient.com.au> --- (In reply to Jonathan Druart from comment #56)

Can we dedicate this bug report to the base idea, then create bug report on top of it for experiment? I'd like to avoid noise for testers and keep the comment list easy to follow.

Sounds like a plan, boss man 👍 -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #105543|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #58 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Remote branch rebased against master, and adjusted. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #60 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to Tomás Cohen Arazi from comment #59)

(In reply to Jonathan Druart from comment #56)

...

Can we dedicate this bug report to the base idea, then create bug report on top of it for experiment? I'd like to avoid noise for testers and keep the comment list easy to follow.

I agree with that.

As discussed by other means, my only concerns are: - should we make it opt-in at this stage (i.e. keep the current scripts, but have a config/syspref that defines if the rabbitmq + daemon should be used). As it is packaged in the supported distros/versions it is not *that* important but worth discussing.

We could, but really I don't think it's worth the effort. We will have to maintain 2 different codes that do the same thing, and will slow down for sure the move for other scripts.

- I don't agree with pulling the rabbitmq-server package automatically on a Debian install. We don't do that with mysql-server/mariadb-server and we should follow that approach. i.e. on the install notes, there will be an entry, before installing koha-common, which will say 'RabbitMQ' and will explain that if you already have one running, you point to it when calling koha-create (bug 25674) and otherwise what command to run for installing it.

Yes sure, I agree. I will implement that as soon as this is PQA. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- URL|https://gitlab.com/joubu/Ko |https://gitlab.com/joubu/Ko |ha/commits/bug_22417 |ha/commits/bug_22417 |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=22417#c30 | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #62 from David Cook <dcook@prosentient.com.au> --- (In reply to Jonathan Druart from comment #60)

Yes sure, I agree. I will implement that as soon as this is PQA.

Could you squash your commits and then post a patch here? I have directly fetched from your Gitlab during testing, but it does make testing and patch evaluation more difficult. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #64 from David Cook <dcook@prosentient.com.au> --- (In reply to David Cook from comment #62)

(In reply to Jonathan Druart from comment #60)

...

Yes sure, I agree. I will implement that as soon as this is PQA.

Could you squash your commits and then post a patch here?

If you squash your commits and post a patch here, I'd be happy to collaborate on this work. As you know, I've already got a similar implementation running locally for 1 Koha instance, and I am very keen to move forward with a community implementation instead. I also have a Koha Plugin in development which I would love to use with a message queue. Between Jonathan, Tomas, and David, I'm sure we can make this happen soon. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |ASSIGNED -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #67 from Tomás Cohen Arazi <tomascohen@gmail.com> --- I would like my signature to be added to this work. I think there are a couple areas we need to work on: - Tasks classes organization (I don't really like them much) - Plugins: how they fit here Those two can be dealt with later, and I'm sure it is better to have this pushed ASAP to address any integration issues sooner than later. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #68 from David Cook <dcook@prosentient.com.au> --- (In reply to Tomás Cohen Arazi from comment #67)

I'm sure it is better to have this pushed ASAP to address any integration issues sooner than later.

Sounds good enough to me. I think that the most important thing is to get the Net::Stomp dependency added, and a configuration mechanism so that Koha code can get connection details for a RabbitMQ instance. With that in place, we should be able to refine and polish the overall task queue feature incrementally. As you say, Tomas, the sooner we get this merged into Koha, the better. (By extension, I'm already locally using RabbitMQ with Koha, but I'm using a custom block in koha-conf.xml and custom update to debian/control to do that on the Koha side. It would be great for those to be provided by upstream. That would free me up to focus on my custom RabbitMQ consumer and my custom Koha producer code. Once the task queue feature is optimized, I could replace most of that custom work too, other than whatever "driver" is being used in the Koha RabbitMQ consumer.) -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #70 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 107386 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=107386&action=edit Bug 22417: DBIX Schema changes Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #72 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 107388 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=107388&action=edit Bug 22417: Add Koha::BackgroundJob::BatchUpdateBiblio Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #74 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 107390 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=107390&action=edit Bug 22417: [DO NOT PUSH] Add simple worker and add_job scripts Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #76 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 107392 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=107392&action=edit Bug 22417: Adapt the batch_record_modification tool Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #78 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 107394 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=107394&action=edit Bug 22417: Fix borrowernumber values Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #80 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 107396 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=107396&action=edit Bug 22417: Switch to STOMP apt install rabbitmq-server service rabbitmq-server start rabbitmq-plugins enable rabbitmq_stomp apt install libnet-stomp-perl cp debian/scripts/koha-functions.sh /usr/share/koha/bin/koha-functions.sh cp debian/scripts/koha-worker /usr/bin/ koha-worker --start kohadev tail -f /var/log/rabbitmq/* Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #82 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 107398 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=107398&action=edit Bug 22417: Try to display pending jobs - only work if worker not started ie. no subscription yet This needs to be fixed (if possible), or removed from the patchset Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #84 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 107400 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=107400&action=edit Bug 22417: Restore the concept of namespace Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #86 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 107402 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=107402&action=edit Bug 22417: Add a note about the existence of the DB row Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #88 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 107404 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=107404&action=edit Bug 22417: Restore the 'add to list' feature This feature has been added recently by bug 18127. It highlights the need of a post processing hook. Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #90 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 107406 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=107406&action=edit Bug 22417: Send message to /queue This cannot be: /queue/$namespace/$job_type I got: '/koha_kohadev/batch_biblio_record_modification' is not a valid queue destination So keeping the dash in $namespace-$job_type Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #92 from David Cook <dcook@prosentient.com.au> --- (In reply to Jonathan Druart from comment #89)

Created attachment 107405 [details] [review] Bug 22417: Add the new permission manage_background_jobs

QA: Please answer the question in admin/background_jobs.pl

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

I can't 100% recall how background jobs are implemented in the web UI, but I find admin/background_jobs.pl to be a challenging topic. In theory, any module (eg circ, acquisitions, etc) could use a background job, so shouldn't the UI be in the source module rather than in the admin module? (If that already is the case and I'm mistaken, then maybe just leave this script with "manage_background_jobs".) -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #94 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 107482 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=107482&action=edit Bug 22417: (follow-up) Fix the batch authority tool -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #95 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to David Cook from comment #92)

(In reply to Jonathan Druart from comment #89)

...

Created attachment 107405 [details] [review] [review] Bug 22417: Add the new permission manage_background_jobs

QA: Please answer the question in admin/background_jobs.pl

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

I can't 100% recall how background jobs are implemented in the web UI, but I find admin/background_jobs.pl to be a challenging topic.

In theory, any module (eg circ, acquisitions, etc) could use a background job, so shouldn't the UI be in the source module rather than in the admin module? (If that already is the case and I'm mistaken, then maybe just leave this script with "manage_background_jobs".)

What is the "source module"? I think that a user with manage_background_jobs should be able to access and manage (cancel, view, etc.) any jobs (done) However anyone who started a job should be allowed to manage this job (not done yet), but should we force a minimum permission or authentication is enough? If we agree on that it can be implemented easily. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #97 from David Cook <dcook@prosentient.com.au> --- I'm finding the number of patches to be a bit intimidating, so I made this little git diff of changed files: Koha/BackgroundJob.pm Koha/BackgroundJob/BatchUpdateAuthority.pm Koha/BackgroundJob/BatchUpdateBiblio.pm Koha/BackgroundJobs.pm Koha/Schema/Result/BackgroundJob.pm admin/background_jobs.pl debian/docs/koha-common.xml debian/koha-common.init debian/scripts/koha-create debian/scripts/koha-disable debian/scripts/koha-functions.sh debian/scripts/koha-remove debian/scripts/koha-worker installer/data/mysql/atomicupdate/bug_15032.perl installer/data/mysql/kohastructure.sql installer/data/mysql/userpermissions.sql koha-tmpl/intranet-tmpl/prog/en/includes/permissions.inc koha-tmpl/intranet-tmpl/prog/en/modules/admin/background_jobs.tt koha-tmpl/intranet-tmpl/prog/en/modules/tools/batch_record_modification.tt koha_worker.pl misc/background_jobs_worker.pl new_koha_job.pl tools/batch_record_modification.pl

From a staff interface perspective, it looks like the main things to test are admin/background_jobs.pl and tools/batch_record_modification.pl?

From a sysadmin perspective, it looks like it would be a good idea to test /etc/init.d/koha-common, koha-create, koha-disable, koha-remove, and koha-worker.

I'm not seeing any usage for "koha_worker.pl" or "new_koha_job.pl". Are these dev scripts that should be removed? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #99 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to David Cook from comment #96)

(In reply to Jonathan Druart from comment #95)

...

I think that a user with manage_background_jobs should be able to access and manage (cancel, view, etc.) any jobs (done) However anyone who started a job should be allowed to manage this job (not done yet), but should we force a minimum permission or authentication is enough?

If we agree on that it can be implemented easily.

I'm not sure I understand 100% what you're saying. I think that I agree that a high level administrator should be able to manage all/any jobs. So manage_background_jobs.pl probably should have a particular/specific admin permission I would think.

That's how it works already, there is a manage_background_jobs permission. The question is about a "normal" user (without that permission) who enqueued a new job. (In reply to David Cook from comment #97)

I'm finding the number of patches to be a bit intimidating, so I made this little git diff of changed files:

Then squash them locally :)

From a staff interface perspective, it looks like the main things to test are admin/background_jobs.pl and tools/batch_record_modification.pl?

From a sysadmin perspective, it looks like it would be a good idea to test /etc/init.d/koha-common, koha-create, koha-disable, koha-remove, and koha-worker.

I'm not seeing any usage for "koha_worker.pl" or "new_koha_job.pl". Are these dev scripts that should be removed?

"Bug 22417: [DO NOT PUSH] Add simple worker and add_job scripts" Those are for tests and won't be pushed. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #101 from David Cook <dcook@prosentient.com.au> --- What's the status of this bug report? It's signed off, but is it waiting for QA or is it still a work in progress? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #103 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to David Cook from comment #101)

What's the status of this bug report? It's signed off, but is it waiting for QA or is it still a work in progress?

It's "Signed Off" and so ready for QA. Why are you asking? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #105 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- You can still add your signed-off-by stamp on the patch set to tell QA you agree with it. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #107 from David Cook <dcook@prosentient.com.au> --- Consider reading my (short) comment https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=1993#c54 It might be better to have Koha code enqueue to a destination of "/koha_kohadev_schedule" and let the message contain the exact details. Then a scheduler (in this case it can just schedule for right now) will take that message and send it (now or in the future at a designated time) to the job-specific queue "/koha_kohadev_batch_biblio_record_modification", which will be read by workers subscribed to that queue. I don't know how big of a change it would be at this point, but it might add some future proofing for when we do have a task scheduler... -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #109 from David Cook <dcook@prosentient.com.au> --- Ah, it looks like I should've ran "restart_all" as well. My bad. 10. Go to http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl?op=view&id=3 11. See Progress 1/10 12. Refresh page after a few seconds 13. See Progress 10/10 14. http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 15. See list of finished jobs Hmm... I'm not seeing any changes to my files... I'll try a less obscure field like 500$a -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #111 from David Cook <dcook@prosentient.com.au> --- Ah, the root user in koha-testing-docker has a HOSTNAME=e507f05ef96a in my case. This works: root@kohadevbox:koha(bug_22417)$ HOSTNAME=kohadevbox rabbitmqctl list_queues Listing queues ... koha_kohadev-batch_authority_record_modification 0 koha_kohadev-batch_biblio_record_modification 0 batch_biblio_record_modification 0 I'm curious about that last queue there, but not a blocker. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #107386|0 |1 is obsolete| | --- Comment #113 from David Cook <dcook@prosentient.com.au> --- Created attachment 108024 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=108024&action=edit Bug 22417: DBIX Schema changes Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: David Cook <dcook@prosentient.com.au> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #107388|0 |1 is obsolete| | --- Comment #115 from David Cook <dcook@prosentient.com.au> --- Created attachment 108026 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=108026&action=edit Bug 22417: Add Koha::BackgroundJob::BatchUpdateBiblio Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: David Cook <dcook@prosentient.com.au> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #107390|0 |1 is obsolete| | --- Comment #117 from David Cook <dcook@prosentient.com.au> --- Created attachment 108028 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=108028&action=edit Bug 22417: [DO NOT PUSH] Add simple worker and add_job scripts Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: David Cook <dcook@prosentient.com.au> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #107392|0 |1 is obsolete| | --- Comment #119 from David Cook <dcook@prosentient.com.au> --- Created attachment 108030 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=108030&action=edit Bug 22417: Adapt the batch_record_modification tool Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: David Cook <dcook@prosentient.com.au> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #107394|0 |1 is obsolete| | --- Comment #121 from David Cook <dcook@prosentient.com.au> --- Created attachment 108032 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=108032&action=edit Bug 22417: Fix borrowernumber values Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: David Cook <dcook@prosentient.com.au> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #107396|0 |1 is obsolete| | --- Comment #123 from David Cook <dcook@prosentient.com.au> --- Created attachment 108034 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=108034&action=edit Bug 22417: Switch to STOMP apt install rabbitmq-server service rabbitmq-server start rabbitmq-plugins enable rabbitmq_stomp apt install libnet-stomp-perl cp debian/scripts/koha-functions.sh /usr/share/koha/bin/koha-functions.sh cp debian/scripts/koha-worker /usr/bin/ koha-worker --start kohadev tail -f /var/log/rabbitmq/* Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: David Cook <dcook@prosentient.com.au> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #107398|0 |1 is obsolete| | --- Comment #125 from David Cook <dcook@prosentient.com.au> --- Created attachment 108036 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=108036&action=edit Bug 22417: Try to display pending jobs - only work if worker not started ie. no subscription yet This needs to be fixed (if possible), or removed from the patchset Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: David Cook <dcook@prosentient.com.au> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #107400|0 |1 is obsolete| | --- Comment #127 from David Cook <dcook@prosentient.com.au> --- Created attachment 108038 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=108038&action=edit Bug 22417: Restore the concept of namespace Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: David Cook <dcook@prosentient.com.au> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #107402|0 |1 is obsolete| | --- Comment #129 from David Cook <dcook@prosentient.com.au> --- Created attachment 108040 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=108040&action=edit Bug 22417: Add a note about the existence of the DB row Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: David Cook <dcook@prosentient.com.au> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #107404|0 |1 is obsolete| | --- Comment #131 from David Cook <dcook@prosentient.com.au> --- Created attachment 108042 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=108042&action=edit Bug 22417: Restore the 'add to list' feature This feature has been added recently by bug 18127. It highlights the need of a post processing hook. Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: David Cook <dcook@prosentient.com.au> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #107406|0 |1 is obsolete| | --- Comment #133 from David Cook <dcook@prosentient.com.au> --- Created attachment 108044 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=108044&action=edit Bug 22417: Send message to /queue This cannot be: /queue/$namespace/$job_type I got: '/koha_kohadev/batch_biblio_record_modification' is not a valid queue destination So keeping the dash in $namespace-$job_type Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: David Cook <dcook@prosentient.com.au> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #107482|0 |1 is obsolete| | --- Comment #135 from David Cook <dcook@prosentient.com.au> --- Created attachment 108046 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=108046&action=edit Bug 22417: (follow-up) Fix the batch authority tool Signed-off-by: David Cook <dcook@prosentient.com.au> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #137 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to David Cook from comment #108)

Used this test plan for server setup: 1. apt install rabbitmq-server 2. service rabbitmq-server start 3. rabbitmq-plugins enable rabbitmq_stomp 4. apt install libnet-stomp-perl 5. cp debian/scripts/koha-functions.sh /usr/share/koha/bin/koha-functions.sh 6. cp debian/scripts/koha-worker /usr/bin/ 7. koha-worker --start kohadev 8. tail -f /var/log/rabbitmq/*

5. Got this error message: Cannot enqueue this job. (The error was: {UNKNOWN}: Failed to connect: Error connecting to localhost:61613: Connection refused at /usr/share/perl5/Net/Stomp.pm line 27.; giving up at /usr/share/perl5/Net/Stomp.pm line 27. at /kohadevbox/koha/Koha/BackgroundJob.pm line 61 . See the Koha logfile for more information). 6. service rabbitmq-server restart

Yes I noticed that the last time I tried. It needs a restart after we enable the stomp plugin. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #139 from David Cook <dcook@prosentient.com.au> --- Oh, I just remembered something else that should be addressed. The Koha::BackgroundJob->connect method should use configurable options rather than hard-coded default values. (Maybe it would even be OK to use the hard-coded default values in the absence of configured values for usability...) I would also suggest adding "host" to the the $stomp->connect call. Even if we default to "/", it would be nice to allow sysadmins to determine what virtual host to use in RabbitMQ, and it would cost us nothing but a line or two of code/config. (Note that doing this will help the koha-testing-docker and koha-docker projects to run RabbitMQ in a separate container.) -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #141 from David Cook <dcook@prosentient.com.au> --- (In reply to Jonathan Druart from comment #140)

See bug 25674.

Considering I wrote the majority of the content on bug 25674, I'm not 100% sure what you're implying. I'm guessing that you're saying we can fix that later. I won't disagree on that. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #143 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to David Cook from comment #142)

At this point, are we just waiting for the QA team to look at this report?

See comment 103 and comment 104 -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #145 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to David Cook from comment #144)

I figured that its status as an RM priority would bump it up the QA queue

Me too! -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #147 from David Cook <dcook@prosentient.com.au> --- (In reply to Katrin Fischer from comment #146)

Me too? This one isn't for me tho.

Is that due to any particular reason? I'm happy to provide more information if that will help. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=10163 -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #149 from David Cook <dcook@prosentient.com.au> --- I'll let Jonathan respond to the notes, but I agree with your comment, Kyle. I figure with that little UI tidy up, it should be pretty much good to go. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #151 from David Cook <dcook@prosentient.com.au> --- (In reply to Marcel de Rooy from comment #150)

Looking here too now

Yay!!! :D -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #153 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 109270 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=109270&action=edit Bug 22417: Remove useless use statement -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #155 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to Kyle M Hall from comment #148)

This looks really great! There are some rough edges that will need polished, but that doesn't necessarily need to happen right now on this bug.

A few notes:

* Why are koha_worker.pl and new_koha_job.pl in root dir?

* Why does new_koha_job.pl use Koha::BackgroundJob::BatchUpdateBiblio but not Koha::BackgroundJob::BatchUpdateAuthority? Is this needed

As Marcel said, those 2 scripts are for testing only. They won't be pushed.

* use Scalar::Util qw( weaken ); not used

I think at this point, the focus could be on cleaning up the ui and not refactoring the internals until they need to be refactored.

The most glaring issue is this message presented to the user: The job has been enqueued! It will be processed as soon as possible (FIXME - well, it could depend on a config?)

Fixed. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #156 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- QA Comment: Great to see us moving to a message queue in Koha ! Nice start. And surely things to discuss ;) The UI changes are not really the most relevant imo. We could start as an experimental feature too if needed (in that case we should not touch batch mod). But we should add imo some configuration options (see below, setting up vhost, user, etc.) And perhaps even more important and already commented on a bit, we are adding a task queue here with a message queue. Obviously, we could send other stuff into the MQ; so this might become a problem later on. And sounding a bit nasty, but not meant to be: We are saving the data into a background job table as well as sending it to MQ. Aren't we being redundant here? Actually, it looks like our background job could just hook to the background jobs table and process what is left there. So what is the real benefit here in this implementation of the MQ exactly? Detailed comments: my $stomp = Net::Stomp->new( { hostname => 'localhost', port => '61613' } ); hardcoded port number / we should probably also let rabbitmq run in its own container, so replace localhost too: move to koha-conf No disconnect ? Memory issues when not doing so? $stomp->connect( { login => 'guest', passcode => 'guest' } ); hmm; isnt this the place for vhosts to come in ? just as vhost, a user should have been created? / hardcoded: move to koha-conf or separate conf? # Picking a random id (memcached_namespace) from the config my $namespace = C4::Context->config('memcached_namespace'); Shouldnt we configure a koha vhost during install (koha-create) and indeed pick that from koha-conf or so ? Why not use config/database instead? Normally it looks like koha_instance Not sure btw if using the term namespace is confusing here; or just a queue prefix ? Add Koha::BackgroundJob::BatchUpdateBiblio Not sure if we need that. The crux is here only four lines: my $record = C4::Biblio::GetMarcBiblio({ biblionumber => $biblionumber }); C4::MarcModificationTemplates::ModifyRecordWithTemplate( $mmtid, $record ); my $frameworkcode = C4::Biblio::GetFrameworkCode( $biblionumber ); C4::Biblio::ModBiblio( $record, $biblionumber, $frameworkcode ); The rest of the code is more or less overhead which perhaps better could be moved to the general module. The module should care for iterating too. So if we could pass a sub or method to a general process method along with a params hash here including biblionumber and marc_template_id, it might be enough? Similar remarks for enqueue. Would reduce a number of submodules. So we should enqueue: Koha::Backgroundjob->new({ type => 'batch_record_mod', ... })->enqueue({ mmtid => $mmtid, records => $records }) And process with Koha::Backgroundjobs->process({ type => ... }) where the process method iterates over the records and associates type with e.g. Koha::Biblios::batch_update [four lines above] etc. installer/data/mysql/atomicupdate/bug_15032.perl Wrong bug number diff koha_worker.pl misc/background_jobs_worker.pl Why did you add koha_worker.pl? We already run the misc script ? No tests anywhere [yet]? Ok :) -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #158 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Thanks Marcel and Kyle for having a look at this! (In reply to Marcel de Rooy from comment #156)

QA Comment: Great to see us moving to a message queue in Koha ! Nice start. And surely things to discuss ;) The UI changes are not really the most relevant imo. We could start as an experimental feature too if needed (in that case we should not touch batch mod). But we should add imo some configuration options (see below, setting up vhost, user, etc.) And perhaps even more important and already commented on a bit, we are adding a task queue here with a message queue. Obviously, we could send other stuff into the MQ; so this might become a problem later on. And sounding a bit nasty, but not meant to be: We are saving the data into a background job table as well as sending it to MQ. Aren't we being redundant here? Actually, it looks like our background job could just hook to the background jobs table and process what is left there. So what is the real benefit here in this implementation of the MQ exactly?

I am not sure I understand what you are suggesting. Do you mean misc/background_jobs_worker.pl could watch the DB table and so we could remove the RabbitMQ dependency?

Detailed comments: my $stomp = Net::Stomp->new( { hostname => 'localhost', port => '61613' } ); hardcoded port number / we should probably also let rabbitmq run in its own container, so replace localhost too: move to koha-conf No disconnect ? Memory issues when not doing so? $stomp->connect( { login => 'guest', passcode => 'guest' } ); hmm; isnt this the place for vhosts to come in ? just as vhost, a user should have been created? / hardcoded: move to koha-conf or separate conf?

If we need to tweak the configuration it could go into koha-conf. As I said in a comment (Koha::BackgroundJob->enqueue), specific vhosts need to be defined server-side. Here we just want to have it working out of the box. We will have to focus specifically on that depending on the needs we found once it's pushed.

# Picking a random id (memcached_namespace) from the config my $namespace = C4::Context->config('memcached_namespace'); Shouldnt we configure a koha vhost during install (koha-create) and indeed pick that from koha-conf or so ? Why not use config/database instead? Normally it looks like koha_instance

As Kyle said, it does not really matter for now. We just need an ID.

Not sure btw if using the term namespace is confusing here; or just a queue prefix ?

Maybe you are right but it does not matter much :)

Add Koha::BackgroundJob::BatchUpdateBiblio Not sure if we need that. The crux is here only four lines: my $record = C4::Biblio::GetMarcBiblio({ biblionumber => $biblionumber }); C4::MarcModificationTemplates::ModifyRecordWithTemplate( $mmtid, $record ); my $frameworkcode = C4::Biblio::GetFrameworkCode( $biblionumber ); C4::Biblio::ModBiblio( $record, $biblionumber, $frameworkcode ); The rest of the code is more or less overhead which perhaps better could be moved to the general module. The module should care for iterating too. So if we could pass a sub or method to a general process method along with a params hash here including biblionumber and marc_template_id, it might be enough? Similar remarks for enqueue. Would reduce a number of submodules. So we should enqueue: Koha::Backgroundjob->new({ type => 'batch_record_mod', ... })->enqueue({ mmtid => $mmtid, records => $records }) And process with Koha::Backgroundjobs->process({ type => ... }) where the process method iterates over the records and associates type with e.g. Koha::Biblios::batch_update [four lines above] etc.

Yes, maybe. The idea was to be flexible in case we need it. When we will have more job moved to this mechanism we will know what can be refactored. Also I think it's good to have more code in the module than in the controller scripts.

installer/data/mysql/atomicupdate/bug_15032.perl Wrong bug number

Indeed, it's actually from 15032. It has a long history :)

diff koha_worker.pl misc/background_jobs_worker.pl Why did you add koha_worker.pl? We already run the misc script ?

new_koha_job.pl and koha_worker.pl won't be pushed. They are there for testing purpose only.

No tests anywhere [yet]? Ok :)

For point. My (very) bad tentative to justify would be that existing code was not covered by tests either. Argument rejected. I could try to write some tests for the 2 ->process method, but I don't think ->enqueue or ->connect can be tested easily. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #160 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to Jonathan Druart from comment #158)

(In reply to Marcel de Rooy from comment #156)

...

And perhaps even more important and already commented on a bit, we are adding a task queue here with a message queue. Obviously, we could send other stuff into the MQ; so this might become a problem later on. And sounding a bit nasty, but not meant to be: We are saving the data into a background job table as well as sending it to MQ. Aren't we being redundant here? Actually, it looks like our background job could just hook to the background jobs table and process what is left there. So what is the real benefit here in this implementation of the MQ exactly?

I am not sure I understand what you are suggesting. Do you mean misc/background_jobs_worker.pl could watch the DB table and so we could remove the RabbitMQ dependency?

No, that is not my suggestion here. But this argument is about its architecture. We use a message queue for a more specific goal. This has translated into the actual code making it harder to widen the scope again. Later on, I want to add messages (no tasks) in another area but what if I dont want to go via BackgroundJob? The last few sentences actually ask for a justification why send we the same data to both a table and the message queue. What is in this specific case the added value of the message queue? If we only use it for this specific case, it looks like much overhead for what it brings. If the justification is in the wider use of a message queue, lets translate that to its design. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #162 from David Cook <dcook@prosentient.com.au> --- (In reply to David Cook from comment #161)

With the message queue, workers use a select(2) syscall to wait until they have data to read on the socket connection they have to the message broker.

That is workers that are using Net::Stomp, as it uses a select-based event loop. There's no reason a person couldn't have workers (ie queue consumers) written using different event systems. I think with AnyEvent::STOMP you can specify your own event loop (like the more efficient epoll), or even use a different programming language all together. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #164 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- David answered perfectly to the need of the insert in the table. I will answer about the design then, Koha::BackgroundJob is coming from bug 15032, written more than 3 years ago. The idea here is to provide a fix for the background jobs to make all the controllers work under plack (and so fix the different bugs linked with bug 15032). Nothing else, but it let the ground open to other enhancements. I don't think it's a good idea to block this development more. IMO we really need it into master to get new ideas on top of it. I think we could easily provide a follow-up bug report renaming background job with task queue (DB table, scripts and modules). -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #166 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to Jonathan Druart from comment #164)

David answered perfectly to the need of the insert in the table.

I think we could easily provide a follow-up bug report renaming background job with task queue (DB table, scripts and modules).

Which was not the main point at all. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Signed Off -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #108024|0 |1 is obsolete| | --- Comment #170 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 109320 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=109320&action=edit Bug 22417: DBIX Schema changes Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: David Cook <dcook@prosentient.com.au> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #108026|0 |1 is obsolete| | --- Comment #172 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 109322 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=109322&action=edit Bug 22417: Add Koha::BackgroundJob::BatchUpdateBiblio Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: David Cook <dcook@prosentient.com.au> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #108028|0 |1 is obsolete| | --- Comment #174 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 109324 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=109324&action=edit Bug 22417: [DO NOT PUSH] Add simple worker and add_job scripts Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: David Cook <dcook@prosentient.com.au> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #108030|0 |1 is obsolete| | --- Comment #176 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 109326 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=109326&action=edit Bug 22417: Adapt the batch_record_modification tool Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: David Cook <dcook@prosentient.com.au> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #108032|0 |1 is obsolete| | --- Comment #178 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 109328 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=109328&action=edit Bug 22417: Fix borrowernumber values Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: David Cook <dcook@prosentient.com.au> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #108034|0 |1 is obsolete| | --- Comment #180 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 109330 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=109330&action=edit Bug 22417: Switch to STOMP apt install rabbitmq-server service rabbitmq-server start rabbitmq-plugins enable rabbitmq_stomp apt install libnet-stomp-perl cp debian/scripts/koha-functions.sh /usr/share/koha/bin/koha-functions.sh cp debian/scripts/koha-worker /usr/bin/ koha-worker --start kohadev tail -f /var/log/rabbitmq/* Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: David Cook <dcook@prosentient.com.au> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.