https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35092 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |32572 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32572 [Bug 32572] We should have a background queue for each job type -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35092 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |33898 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33898 [Bug 33898] background_jobs_worker.pl may leave defunct children processes for extended periods of time -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35092 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |32395 CC| |jonathan.druart+koha@gmail. | |com Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32395 [Bug 32395] Why do we send jobs params to the broker -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35092 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on|32395 |31336, 31090, 32342, 31580, | |30654, 30182 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30182 [Bug 30182] background_jobs.status should be an ENUM https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30654 [Bug 30654] Even with RabbitMQ enabled, we should poll the database for jobs at worker startup https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31090 [Bug 31090] Background jobs needs highlights and searchability https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31336 [Bug 31336] In-progress background jobs give incorrect progress message https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31580 [Bug 31580] Review Koha::BackgroundJob->finish and 'failed' case workflow https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32395 [Bug 32395] Why do we send jobs params to the broker -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35092 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david@davidnind.com -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35092 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |32736, 31124, 30656 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30656 [Bug 30656] Introduce worker_manager daemon https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31124 [Bug 31124] koha-remove fails to remove long_tasks queue daemon, so koha-create for same <instance> user fails https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32736 [Bug 32736] koha-worker should be able to restart all queues with a single call -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35092 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on|32342 | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35092 --- Comment #2 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- As a result of the above, I will make 34990 (functionally) dependent of 32305. It is in PQA but should not be pushed separately. Note that we can improve/relocate the code of 32305 but I would recommend to do that while refactoring a bit more. Will still add a comment about options to go further with worker algorithm. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35092 --- Comment #3 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- TODO move the selection to a filter_by method in BackgroundJobs TODO Add a MessageQueue module to move a few methods out of BackgroundJob like connect and .. ? -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35092 --- Comment #5 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Did you see David Cook's findings on the ML? https://lists.koha-community.org/pipermail/koha-devel/2023-October/048379.ht... [Koha-devel] Background jobs quirks -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35092 --- Comment #7 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- We should decide imo how to resolve sync issues as mentioned before. I thought of a few options. [1] Still hybrid but dynamical approach. The worker should regularly check both MQ and DB to select new jobs. So more dynamical than the current approach that is determined by MQ being active or not at the start of the worker. [2] Stricter distinction between MQ and DB: Add a preference like UseMessageQ to indicate that the worker only reads jobs from either MQ or DB. If you choose the DB side, the MQ will no longer be filled. If you choose MQ, the worker should not be able to switch to "DB mode" anymore like it does now. [3] Remove DB mode: This involves "trusting MQ to be stable enough". We will get new jobs from MQ, and only save processed jobs in DB. [4] Remove MQ mode: The other way around. "We do not really trust MQ." Remove enqueuing to MQ and reading new jobs from it. Please give some feedback. What is the best way to go forward? -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35092 --- Comment #9 from Blou <philippe.blouin@inlibro.com> --- I would favor option 4, and option 2 as a lesser evil scenario. We use the database everywhere, we keep trace of everything in it, including deleted objects, and can use it to retrace everything that happened to a library when they ask. And then we use an outside tool for messaging to a background worker coded by/for Koha, for tasks where a nanosecond reaction time is absolutely not necessary. MQ (and Erlang) is surprisingly demanding on ressources for the little it offers. The last two releases, I've hacked the background_jobs tool to make sure the database was always used, and checked at startup, so that no task was lost. You sure can invest a lot of efforts to correct MQ, but why? What is the long term objective for this? Because background tasks are sure not enough to justify the added complexity. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35092 --- Comment #10 from David Cook <dcook@prosentient.com.au> --- (In reply to Marcel de Rooy from comment #7)

We should decide imo how to resolve sync issues as mentioned before. I thought of a few options.

[1] Still hybrid but dynamical approach. The worker should regularly check both MQ and DB to select new jobs. So more dynamical than the current approach that is determined by MQ being active or not at the start of the worker.

[2] Stricter distinction between MQ and DB: Add a preference like UseMessageQ to indicate that the worker only reads jobs from either MQ or DB. If you choose the DB side, the MQ will no longer be filled. If you choose MQ, the worker should not be able to switch to "DB mode" anymore like it does now.

[3] Remove DB mode: This involves "trusting MQ to be stable enough". We will get new jobs from MQ, and only save processed jobs in DB.

[4] Remove MQ mode: The other way around. "We do not really trust MQ." Remove enqueuing to MQ and reading new jobs from it.

Please give some feedback. What is the best way to go forward?

I think option 2 is a reasonable compromise. Personally, my preference is for the MQ. It's a great tool I use for a variety of systems beyond Koha. I understand it well enough that I can investigate and resolve problems encountered while using it. I think the more we use it the more we'll learn and the better we'll become with it, and the further into the present/future we'll move as service providers and technologists. I think it's well known that technology moves quickly and you have to continually learn and adapt with its progress. (Another example is using Vue.js. It's an exciting client side technology that we'd all do well to learn both for the benefit of Koha and ourselves as software professionals.) That said, I can understand that some people are more comfortable using the software tools that they already know and don't want to push boundaries or explore frontiers. There is certainly merit to sticking to battle tested tools that are well understood by a broad base. One of the downsides of option 2 is that it splits focus though. The Koha community has limited resources and part of our success has come from most of us implementing Koha in the same way, as we've been able to work together to advance the project. If we continue to split between MQ and DB, we might find it becomes more difficult to maintain. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35092 --- Comment #12 from David Cook <dcook@prosentient.com.au> --- (In reply to David Cook from comment #11)

I've just reported bug 35111 which I hope will improve the experience of MQ driven background jobs in Koha.

Funny enough the problem isn't with RabbitMQ per se but rather with the background jobs worker crashing because it lost its connection to the MySQL database. Food for thought there.

Actually, after further investigation, it looks like it was even more complicated. It looks like a difference in behaviour of MySQL C client libraries vs MariaDB C client libraries (the former used by Ubuntu and the latter used by Debian). Easy fix in the end anyway... -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35092 --- Comment #14 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to Blou from comment #9)

I would favor option 4, and option 2 as a lesser evil scenario.

Thanks for your feedback. Your point makes sense. The dilemma is that the other side makes sense too imo ;) -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35092 --- Comment #16 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to Jonathan Druart from comment #13)

My proposal: 1. Push bug 35111 (because it might fix a lot of the trouble people encounters in production!) and have it part of 23.11 2. Ask people on the ML what *exact* problems they are getting with MQ and why they are not using it. 3. Write a script to reproduce the problems locally (run as root so that you can restart things): * Fast enqueue of thousands of request * Enqueue huge ES reindex * Deal with disconnects * Other problems with null/0 status * Other problems from the ML 4. Fix the main bugs 5. Clean up 6. Fix the easy bugs

Thanks. Not sure if 35111 will address that. We can try the ML but we should have had that feedback already. Would we receive it now? "Fix the main bugs". We should first prioritize but imo we need to choose a direction there and you do not give your view about that here? -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35092 --- Comment #18 from Blou <philippe.blouin@inlibro.com> --- I'm all for good technologies for solving problems, but I have always had a problem with technology for technology's sake. My personnal empowerment and my resume-building should not be reasons to rearchitecture a product such as Koha. So what is the endgame for MQ ? I made a genuine search for a RFC on MQ (I admit I'm bad, I could have missed it), and looked at the initial bzs, but I couldn't find what was the future use for MQ ? All I see are the background tasks, which is a simple problem with a simple solution. Could someone sell it to me (and others) ? -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35092 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |35655 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35655 [Bug 35655] Make it possible to switch off RabbitMQ without any warns in logs/about page -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35092 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |35819 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35819 [Bug 35819] "No job found" error for BatchUpdateBiblioHoldsQueue (race condition) -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35092 --- Comment #20 from David Cook <dcook@prosentient.com.au> --- (In reply to Blou from comment #18)

Could someone sell it to me (and others) ?

The MQ works pretty much perfectly for me (at least since bug 35111), so I'd need someone to sell me not using the MQ. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35092 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |julian.maurice@biblibre.com --- Comment #22 from Julian Maurice <julian.maurice@biblibre.com> --- (In reply to David Cook from comment #20)

The MQ works pretty much perfectly for me (at least since bug 35111), so I'd need someone to sell me not using the MQ. If it works for you and solves a real problem, then great. But not everyone needs it and making an external dependency like RabbitMQ mandatory should be done with care as it can add a lot of "small costs" everywhere (for devs, testers, server admins, ...).

The current hybrid approach has proven to be bug prone. So it may be hard to balance enhancing master and fixing stable branches.. By "hybrid" do you mean the fact that one script does both things (MQ and SQL

(In reply to Marcel de Rooy from comment #21) polling) ? Would that be better to have two separate "worker" scripts ? -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35092 --- Comment #24 from Blou <philippe.blouin@inlibro.com> --- (In reply to Marcel de Rooy from comment #23)

An attempt to get rid of various out-of-sync problems, race conditions etc. between MQ and DB as we are having now.

We fixed that by getting rid of MQ. Flawless Victory. :^) -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35092 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |35920 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35920 [Bug 35920] Centralize code from workers -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35092 Jonathan Druart <jonathan.druart@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |REOPENED Resolution|DUPLICATE |--- --- Comment #27 from Jonathan Druart <jonathan.druart@gmail.com> --- (In reply to wajasu from comment #26)

Cleaned up this old bug since it was resolved by bug 32465

*** This bug has been marked as a duplicate of bug 32465 ***

This is an omnibus bug and should stay opened until all dependent bugs have been resolved. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35092 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |34596 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34596 [Bug 34596] Items in transit should not show up in the holds queue -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35092 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |36901 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36901 [Bug 36901] background_jobs_worker.pl sets the status to 'failed' right away -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35092 Bug 35092 depends on bug 30654, which changed state. Bug 30654 Summary: Even with RabbitMQ enabled, we should poll the database for jobs at worker startup https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30654 What |Removed |Added ---------------------------------------------------------------------------- Status|In Discussion |RESOLVED Resolution|--- |WONTFIX -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35092 Bug 35092 depends on bug 32305, which changed state. Bug 32305 Summary: Background worker doesn't check job status when received from rabbitmq https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32305 What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to oldstable |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35092 Bug 35092 depends on bug 36901, which changed state. Bug 36901 Summary: Add logging for uncaught exceptions in background job classes https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36901 What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to oldstable |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35092 Bug 35092 depends on bug 35655, which changed state. Bug 35655 Summary: Make it possible to switch off RabbitMQ without any warns in logs/about page https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35655 What |Removed |Added ---------------------------------------------------------------------------- Status|Needs documenting |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35092 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=41506 -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.