[Bug 27783] New: Add handling for 'Queues' in the background job queue
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Bug ID: 27783 Summary: Add handling for 'Queues' in the background job queue Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Architecture, internals, and plumbing Assignee: koha-bugs@lists.koha-community.org Reporter: martin.renvoize@ptfs-europe.com QA Contact: testopia@bugs.koha-community.org Right now our worker will just pick tasks of the queue in a fifo order.. we should really allow the worker to be more specific and have different workers picking jobs from different queues.. and we shouldn't limit ourselves to 'one queue for job type' or 'one queue for all jobs types. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=27344 -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |27344 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 [Bug 27344] Implement Elastic's update_index_background using Koha::BackgroundJob -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #1 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 132642 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132642&action=edit Bug 27783: Replace --job-type by --job-queue This patch adds a new column background_jobs.queue, which default to 'default' By default, new jobs are enqueued into this default queue, and the background job worker will subscribe to the default queue unless told otherwise by the --job-queue option The new job UpdateElasticIndex is automatically enqueued in another queue named 'index' So you can have 'default' worker with misc/background_jobs_worker.pl and a dedicated indexing worker with misc/background_jobs_worker.pl --job-queue index This is to address bug 27344 comment #15 Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #2 from Julian Maurice <julian.maurice@biblibre.com> --- Created attachment 132644 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132644&action=edit Bug 27783: Add missing atomicupdate file -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #3 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 132649 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132649&action=edit Bug 27783: Replace --job-type by --queue This patch adds a new column background_jobs.queue, which default to 'default' By default, new jobs are enqueued into this default queue, and the background job worker will subscribe to the default queue unless told otherwise by the --job-queue option The new job UpdateElasticIndex is automatically enqueued in another queue named 'index' So you can have 'default' worker with misc/background_jobs_worker.pl and a dedicated indexing worker with misc/background_jobs_worker.pl --queue index This is to address bug 27344 comment #15 Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132644|0 |1 is obsolete| | --- Comment #4 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 132650 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132650&action=edit Bug 27783: Add missing atomicupdate file Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #5 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 132651 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132651&action=edit Bug 27783: (follow-up) Add DB change to installer Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132642|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #6 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- OK.. This works, but... We end up without a worker running under default installs that will run the indexing jobs. We should either add another set of debian templates etc to create the new worker along side the existing koha.worker service... Or.. and I think it is perhaps preferable.. we should work on a worker manager daemon script instead of a simple worker script.. such a script would for workers, plural, to run the various queues. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |In Discussion --- Comment #7 from David Cook <dcook@prosentient.com.au> --- I think that a "default" queue is the wrong approach. Tasks should be in "to-purpose" queues, so that they can have "to-purpose" workers. That said, I don't think the "--job-type" style is sustainable/maintainable as it could get unmanageably long. I suggest we use configuration files. Out of the box, workers use "default_worker_conf.json" which contains all the queues (this file is maintained by the koha-common package). This can be customized using a CLI arg like "-c,--config /path/to/custom_worker_conf.json". This way, Koha "just works" out of the box with no sysadmin effort. For more complex implementations, the onus is on sysadmins to update their custom worker configuration(s). -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #8 from David Cook <dcook@prosentient.com.au> --- (In reply to Martin Renvoize from comment #6)
Or.. and I think it is perhaps preferable.. we should work on a worker manager daemon script instead of a simple worker script.. such a script would for workers, plural, to run the various queues.
Good call. For other projects, I use systemd services or separate containers for workers, but with Koha we use the "daemon" utility to run our own services, so I think we need to do a bespoke manager as well... I suppose koha-worker could consume a declarative configuration file that lists out how many workers to create and what configuration files they should use themselves. It would then invoke those using the "daemon" utility I suppose? -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |tomascohen@gmail.com --- Comment #9 from Tomás Cohen Arazi <tomascohen@gmail.com> --- I was reading the code and thinking how plugin-generated tasks could fit here. And with the current approach, I was leaning into adding 'plugin-real-time' and 'plugin-long-running' queues on top of this. But the need to have 4 running workers sounds like too much... What about having: * 'real-time' * 'turtle' It felt like it would be enough. From the plugin standpoint (to be worked on bug 30410), I would add high-level methods for queueing real-time and long-running tasks, and it would be up to the plugin author to call the right one. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #10 from Tomás Cohen Arazi <tomascohen@gmail.com> --- The 'turtle' queue name was put to make it obvious what I meant. But it is cool too <.< -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |30410 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30410 [Bug 30410] Add a way for plugins to register background tasks -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #11 from David Cook <dcook@prosentient.com.au> --- (In reply to Tomás Cohen Arazi from comment #10)
The 'turtle' queue name was put to make it obvious what I meant. But it is cool too <.<
I laughed. I like it haha. (In reply to Tomás Cohen Arazi from comment #9)
I was reading the code and thinking how plugin-generated tasks could fit here. And with the current approach, I was leaning into adding 'plugin-real-time' and 'plugin-long-running' queues on top of this. But the need to have 4 running workers sounds like too much...
You wouldn't need to have 4 running workers though, since a worker can subscribe to multiple queues. Regardless of the implementation, I am keen to get plugin-generated tasks into Koha.
What about having:
* 'real-time' * 'turtle'
Personally, I would prefer more granularity and configurability, but this does sound like a pragmatic compromise. Another benefit of having "real-time" and "turtle" would be that adding more background tasks should have a lower barrier to entry, since you'd just need to provide a class for the worker to use and an interface for enqueuing tasks. (I've been concerned that people haven't been taking advantage of the background tasks because they weren't sure how to work with it.) -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #12 from David Cook <dcook@prosentient.com.au> --- (In reply to David Cook from comment #11)
Another benefit of having "real-time" and "turtle" would be that adding more background tasks should have a lower barrier to entry, since you'd just need to provide a class for the worker to use and an interface for enqueuing tasks. (I've been concerned that people haven't been taking advantage of the background tasks because they weren't sure how to work with it.)
Admittedly, I've wanted to use the background tasks for local customizations, but the current amount of customization required of core Koha::BackgroundJob code has made me avoid it. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #13 from Tomás Cohen Arazi <tomascohen@gmail.com> --- (In reply to David Cook from comment #11)
You wouldn't need to have 4 running workers though, since a worker can subscribe to multiple queues.
Well, that's the point. You need to set special workers using the .pl script instead of koha-worker, etc... so it complicates things. This is what I valued the most when looking at this patchset.
Another benefit of having "real-time" and "turtle" would be that adding more background tasks should have a lower barrier to entry, since you'd just need to provide a class for the worker to use and an interface for enqueuing tasks. (I've been concerned that people haven't been taking advantage of the background tasks because they weren't sure how to work with it.)
Look at the follow-up bugs and bug 30360 for ideas. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also|https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=27344 | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |27344 Depends on|27344 | Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 [Bug 27344] Implement Elastic's update_index_background using Koha::BackgroundJob -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Add handling for 'Queues' |Introduce background job |in the background job queue |queues -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |martin.renvoize@ptfs-europe |ity.org |.com Depends on| |30345 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30345 [Bug 30345] Koha::BackgroundJob->enqueue should set borrowernumber=undef if no userenv -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Discussion |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132649|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132650|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132651|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #14 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 132846 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132846&action=edit Bug 27783: Add --job-type to background_jobs_worker.pl This will let sysop adjust the number of workers and how they want to manage them. For instance one could want to have one worker for ES indexation and another worker for other jobs, to prevent ES index to be stuck behind bigger batch process. Signed-off-by: Arthur Suzuki <arthur.suzuki@biblibre.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #15 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 132847 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132847&action=edit Bug 27783: Replace --job-type by --queue This patch adds a new column background_jobs.queue, which default to 'default' By default, new jobs are enqueued into this default queue, and the background job worker will subscribe to the default queue unless told otherwise by the --job-queue option The new job UpdateElasticIndex is automatically enqueued in another queue named 'index' So you can have 'default' worker with misc/background_jobs_worker.pl and a dedicated indexing worker with misc/background_jobs_worker.pl --queue index This is to address bug 27344 comment #15 Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #16 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 132848 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132848&action=edit Bug 27783: Add missing atomicupdate file Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #17 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 132849 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132849&action=edit Bug 27783: (follow-up) Add DB change to installer Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #18 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 132850 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132850&action=edit Bug 27783: Rename queues and adjust currently defined jobs This patch renames the queues so the default is the **real-time** one, and the other (*turtle*) is kept for **slow** tasks. All current *batch* tasks are explicitly assigned to the **turtle** queue as well. Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #19 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Ok, after several talks at the hackfest, everyone agreed we should (or not opposed to) reverse the dependency to keep things clearer. I took on the task of rebasing, reorganizing things, etc keeping the attribution of the original patches. To put it clear, this *should* only be lacking what it already lacked of: integration with koha-worker so both worker processes are launched by default to .deb install users. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |29346 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29346 [Bug 29346] Add real-time holds queue update option -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132848|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132849|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132850|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132846|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132847|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #20 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 132875 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132875&action=edit Bug 27783: Add --job-type to background_jobs_worker.pl This will let sysop adjust the number of workers and how they want to manage them. For instance one could want to have one worker for ES indexation and another worker for other jobs, to prevent ES index to be stuck behind bigger batch process. Signed-off-by: Arthur Suzuki <arthur.suzuki@biblibre.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #21 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 132876 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132876&action=edit Bug 27783: Replace --job-type by --queue This patch adds a new column background_jobs.queue, which default to 'default' By default, new jobs are enqueued into this default queue, and the background job worker will subscribe to the default queue unless told otherwise by the --job-queue option The new job UpdateElasticIndex is automatically enqueued in another queue named 'index' So you can have 'default' worker with misc/background_jobs_worker.pl and a dedicated indexing worker with misc/background_jobs_worker.pl --queue index This is to address bug 27344 comment #15 Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #22 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 132877 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132877&action=edit Bug 27783: Add missing atomicupdate file Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #23 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 132878 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132878&action=edit Bug 27783: (follow-up) Add DB change to installer Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #24 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 132879 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132879&action=edit Bug 27783: Rename queues and adjust currently defined jobs This patch renames the queues so the default is the **real-time** one, and the other (*long_tasks*) is kept for **long running** tasks. All current *batch* tasks are explicitly assigned to the **long_tasks** queue as well. Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off --- Comment #25 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Somehow, my working branch was missing 30172, so I rebased this patches on top of it. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #26 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 132880 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132880&action=edit Bug 27783: (QA follow-up) Make dbrev idempotent Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |additional_work_needed -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #27 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 132948 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132948&action=edit Bug 27783: DBIC update [DO NOT PUSH] Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #28 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 132950 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132950&action=edit Bug 27783: Add queue handling in packaging tools This patch adds queue name handling in is_worker_running and a handy method to centralize process name handling for queue-specific workers. To test: 1. Apply this patch 2. Run: $ source debian/scripts/koha-functions.sh $ get_worker_name kohadev kohadev-koha-worker => SUCCESS: Default queue means no queue name included $ get_worker_name kohadev default kohadev-koha-worker => SUCCESS: Default queue means no queue name included $ get_worker_name kohadev long_tasks kohadev-koha-worker-long_tasks => SUCCESS: Queue name appended to the name 4. Run: $ perl misc4dev/cp_debian_files.pl \ --koha_dir /kohadevbox/koha \ --gitify /kohadevbox/gitify \ --instance kohadev 5. Run: $ koha-worker --start --queue oleonard kohadev => SUCCESS: Starts! 6. Check: $ ps waux | grep oleonard => SUCCESS: It is running 7. Play with --stop, --restart, --status and different queue names, including 'default' and without the --queue param as well. => SUCCESS: All as expected! 8. Sign off :-D Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132950|0 |1 is obsolete| | --- Comment #29 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 132951 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132951&action=edit Bug 27783: Add queue handling in packaging tools This patch adds queue name handling in is_worker_running and a handy method to centralize process name handling for queue-specific workers. To test: 1. Apply this patch 2. Run: $ source debian/scripts/koha-functions.sh $ get_worker_name kohadev kohadev-koha-worker => SUCCESS: Default queue means no queue name included $ get_worker_name kohadev default kohadev-koha-worker => SUCCESS: Default queue means no queue name included $ get_worker_name kohadev long_tasks kohadev-koha-worker-long_tasks => SUCCESS: Queue name appended to the name 4. Run: $ perl misc4dev/cp_debian_files.pl \ --koha_dir /kohadevbox/koha \ --gitify /kohadevbox/gitify \ --instance kohadev 5. Run: $ koha-worker --start --queue oleonard kohadev => SUCCESS: Starts! 6. Check: $ ps waux | grep oleonard => SUCCESS: It is running 7. Play with --stop, --restart, --status and different queue names, including 'default' and without the --queue param as well. => SUCCESS: All as expected! 8. Sign off :-D Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|additional_work_needed | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |julian.maurice@biblibre.com | |, | |kyle@bywatersolutions.com, | |nick@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart+koha@gmail. | |com --- Comment #30 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- 1. Shouldn't we force the job to define its queue? Otherwise I have the feeling that we will end up with "default" everywhere. 2. What could be other queues? 3. Is BatchCancelHold really a long task? How do you define a short/long task then? 4. Do we really want long/short task distinction? If you index 1000 records (using batch update), it will be considered as "default" (/not long task), but it will take a while. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #31 from Tomás Cohen Arazi <tomascohen@gmail.com> --- (In reply to Jonathan Druart from comment #30)
1. Shouldn't we force the job to define its queue? Otherwise I have the feeling that we will end up with "default" everywhere.
I think this can be caught at QA time. See 4.
2. What could be other queues?
Chatter on this and bug 27344 pointed to devs wantings some flexibility to run their own queues. That's why I reorganized things, keeping that in mind. Maybe later someone adds a configuration page/file to point specific tasks to specific queues, in order to offload some tasks to a separate physical server.
3. Is BatchCancelHold really a long task? How do you define a short/long task then?
I think that was a mistake I made. i.e. cancelling holds shouldn't wait for a long running batch import to finish.
4. Do we really want long/short task distinction? If you index 1000 records (using batch update), it will be considered as "default" (/not long task), but it will take a while.
We do want that. And maybe in the future we can add more queues... and priorities. I think a follow-up is needed in bug 27344 to make it resort to long_tasks if the record count is high (for example, because of a batch operation) and be real-time if we are just updating an item index because it's been checked-in. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #32 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- (In reply to Tomás Cohen Arazi from comment #31)
(In reply to Jonathan Druart from comment #30)
1. Shouldn't we force the job to define its queue? Otherwise I have the feeling that we will end up with "default" everywhere.
I think this can be caught at QA time. See 4.
We have job_type that is doing that already. Why asking a human to catch something that can be caught by a machine? :) 170 sub job_type { croak "This method must be subclassed" } -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #33 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- I am not sure I am understanding what bring 'queue' that cannot be done with the job_type. We could have --job-type and a negation for this option. perl background_jobs_worker.pl --job-type update_elastic_index => Run only indexation tasks perl background_jobs_worker.pl --not-job-type update_elastic_index => Run all but indexation tasks -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #34 from David Cook <dcook@prosentient.com.au> --- (In reply to Jonathan Druart from comment #33)
I am not sure I am understanding what bring 'queue' that cannot be done with the job_type.
We could have --job-type and a negation for this option.
perl background_jobs_worker.pl --job-type update_elastic_index => Run only indexation tasks
perl background_jobs_worker.pl --not-job-type update_elastic_index => Run all but indexation tasks
I think that would get really messy/confusing as the number of job types/queues grows. I still think pointing background_jobs_worker.pl at a configuration file makes the most sense. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #35 from Tomás Cohen Arazi <tomascohen@gmail.com> --- (In reply to David Cook from comment #34)
(In reply to Jonathan Druart from comment #33)
I am not sure I am understanding what bring 'queue' that cannot be done with the job_type.
We could have --job-type and a negation for this option.
perl background_jobs_worker.pl --job-type update_elastic_index => Run only indexation tasks
perl background_jobs_worker.pl --not-job-type update_elastic_index => Run all but indexation tasks
I think that would get really messy/confusing as the number of job types/queues grows.
I still think pointing background_jobs_worker.pl at a configuration file makes the most sense.
Right, as the job types grows, and we don't even have control of them (plugins) we need some other way of grouping them more generally. Otherwise we cannot do cool things like installing a plugin and it will just work. We would need to set a worker for that. I'm sure you will insist on the ---not use for that, but think how we would solve the real-time vs. no hurry use cases in a generic way. It is plain straight-forward to just 'subscribe to a queue' for authors. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #36 from Tomás Cohen Arazi <tomascohen@gmail.com> --- (In reply to Jonathan Druart from comment #30)
1. Shouldn't we force the job to define its queue? Otherwise I have the feeling that we will end up with "default" everywhere.
I've been trying to think of use cases, and the authority merge came to my mind. If the record count is low enough it would be great to do it real-time (because librarians really want to keep working on those records, probably), and if it is too big, it would be great to do it in the long_tasks queue (because it wouldn't need to wait for the cronjob). So, in this case, ->enqueue could actually put the job in different queues depending on the scenario. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |kyle@bywatersolutions.com |y.org | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132875|0 |1 is obsolete| | Attachment #132876|0 |1 is obsolete| | Attachment #132877|0 |1 is obsolete| | Attachment #132878|0 |1 is obsolete| | Attachment #132879|0 |1 is obsolete| | Attachment #132880|0 |1 is obsolete| | Attachment #132948|0 |1 is obsolete| | Attachment #132951|0 |1 is obsolete| | --- Comment #37 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 133160 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=133160&action=edit Bug 27783: Add --job-type to background_jobs_worker.pl This will let sysop adjust the number of workers and how they want to manage them. For instance one could want to have one worker for ES indexation and another worker for other jobs, to prevent ES index to be stuck behind bigger batch process. Signed-off-by: Arthur Suzuki <arthur.suzuki@biblibre.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #133160|0 |1 is obsolete| | --- Comment #38 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 133161 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=133161&action=edit Bug 27783: Add --job-type to background_jobs_worker.pl This will let sysop adjust the number of workers and how they want to manage them. For instance one could want to have one worker for ES indexation and another worker for other jobs, to prevent ES index to be stuck behind bigger batch process. Signed-off-by: Arthur Suzuki <arthur.suzuki@biblibre.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #39 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 133162 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=133162&action=edit Bug 27783: Replace --job-type by --queue This patch adds a new column background_jobs.queue, which default to 'default' By default, new jobs are enqueued into this default queue, and the background job worker will subscribe to the default queue unless told otherwise by the --job-queue option The new job UpdateElasticIndex is automatically enqueued in another queue named 'index' So you can have 'default' worker with misc/background_jobs_worker.pl and a dedicated indexing worker with misc/background_jobs_worker.pl --queue index This is to address bug 27344 comment #15 Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #40 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 133163 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=133163&action=edit Bug 27783: Add missing atomicupdate file Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #41 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 133164 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=133164&action=edit Bug 27783: (follow-up) Add DB change to installer Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #42 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 133165 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=133165&action=edit Bug 27783: Rename queues and adjust currently defined jobs This patch renames the queues so the default is the **real-time** one, and the other (*long_tasks*) is kept for **long running** tasks. All current *batch* tasks are explicitly assigned to the **long_tasks** queue as well. Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #43 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 133166 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=133166&action=edit Bug 27783: (QA follow-up) Make dbrev idempotent Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #44 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 133167 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=133167&action=edit Bug 27783: DBIC update [DO NOT PUSH] Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #45 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 133168 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=133168&action=edit Bug 27783: Add queue handling in packaging tools This patch adds queue name handling in is_worker_running and a handy method to centralize process name handling for queue-specific workers. To test: 1. Apply this patch 2. Run: $ source debian/scripts/koha-functions.sh $ get_worker_name kohadev kohadev-koha-worker => SUCCESS: Default queue means no queue name included $ get_worker_name kohadev default kohadev-koha-worker => SUCCESS: Default queue means no queue name included $ get_worker_name kohadev long_tasks kohadev-koha-worker-long_tasks => SUCCESS: Queue name appended to the name 4. Run: $ perl misc4dev/cp_debian_files.pl \ --koha_dir /kohadevbox/koha \ --gitify /kohadevbox/gitify \ --instance kohadev 5. Run: $ koha-worker --start --queue oleonard kohadev => SUCCESS: Starts! 6. Check: $ ps waux | grep oleonard => SUCCESS: It is running 7. Play with --stop, --restart, --status and different queue names, including 'default' and without the --queue param as well. => SUCCESS: All as expected! 8. Sign off :-D Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #46 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 133169 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=133169&action=edit Bug 27783: (QA follow-up) Make dbrev executable Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #133161|0 |1 is obsolete| | --- Comment #47 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 133170 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=133170&action=edit Bug 27783: Add --job-type to background_jobs_worker.pl This will let sysop adjust the number of workers and how they want to manage them. For instance one could want to have one worker for ES indexation and another worker for other jobs, to prevent ES index to be stuck behind bigger batch process. Signed-off-by: Arthur Suzuki <arthur.suzuki@biblibre.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #133162|0 |1 is obsolete| | --- Comment #48 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 133171 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=133171&action=edit Bug 27783: Replace --job-type by --queue This patch adds a new column background_jobs.queue, which default to 'default' By default, new jobs are enqueued into this default queue, and the background job worker will subscribe to the default queue unless told otherwise by the --job-queue option The new job UpdateElasticIndex is automatically enqueued in another queue named 'index' So you can have 'default' worker with misc/background_jobs_worker.pl and a dedicated indexing worker with misc/background_jobs_worker.pl --queue index This is to address bug 27344 comment #15 Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #133163|0 |1 is obsolete| | --- Comment #49 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 133172 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=133172&action=edit Bug 27783: Add missing atomicupdate file Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #133164|0 |1 is obsolete| | --- Comment #50 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 133173 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=133173&action=edit Bug 27783: (follow-up) Add DB change to installer Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #133165|0 |1 is obsolete| | --- Comment #51 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 133174 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=133174&action=edit Bug 27783: Rename queues and adjust currently defined jobs This patch renames the queues so the default is the **real-time** one, and the other (*long_tasks*) is kept for **long running** tasks. All current *batch* tasks are explicitly assigned to the **long_tasks** queue as well. Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #133166|0 |1 is obsolete| | --- Comment #52 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 133175 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=133175&action=edit Bug 27783: (QA follow-up) Make dbrev idempotent Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #133167|0 |1 is obsolete| | --- Comment #53 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 133176 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=133176&action=edit Bug 27783: DBIC update [DO NOT PUSH] Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #133168|0 |1 is obsolete| | --- Comment #54 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 133177 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=133177&action=edit Bug 27783: Add queue handling in packaging tools This patch adds queue name handling in is_worker_running and a handy method to centralize process name handling for queue-specific workers. To test: 1. Apply this patch 2. Run: $ source debian/scripts/koha-functions.sh $ get_worker_name kohadev kohadev-koha-worker => SUCCESS: Default queue means no queue name included $ get_worker_name kohadev default kohadev-koha-worker => SUCCESS: Default queue means no queue name included $ get_worker_name kohadev long_tasks kohadev-koha-worker-long_tasks => SUCCESS: Queue name appended to the name 4. Run: $ perl misc4dev/cp_debian_files.pl \ --koha_dir /kohadevbox/koha \ --gitify /kohadevbox/gitify \ --instance kohadev 5. Run: $ koha-worker --start --queue oleonard kohadev => SUCCESS: Starts! 6. Check: $ ps waux | grep oleonard => SUCCESS: It is running 7. Play with --stop, --restart, --status and different queue names, including 'default' and without the --queue param as well. => SUCCESS: All as expected! 8. Sign off :-D Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #133169|0 |1 is obsolete| | --- Comment #55 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 133178 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=133178&action=edit Bug 27783: (QA follow-up) Make dbrev executable Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #56 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 133179 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=133179&action=edit Bug 27783: (follow-up) Rename options Why require `job_`.. it's easiery to just use 'queue' and we have 'job' from the context of the script we're calling. This also inproves consistency between the debian commands and the script params. Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #57 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Just another QA stamp here having re-tested the set. This gives us two workers by default, one working through long-running tasks sequentially and the second working through short running tasks sequentially.. thus we can now work on short and long tasks in parallel and not cause hold-ups with a short running task waiting on a long-running one. This gets us a good way towards where we want to be eventually with a configurable manager daemon with a config file and self spawning workers as Davids config file approach mentioned above suggests.. this is a great interim measure though for now and allows us to continue working on tasks whilst we consider the options for implementing such a manager daemon. (Job dependencies and things will need to be added too to allow for such a manager system to work). -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to master Version(s)| |22.05.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #58 from Fridolin Somers <fridolin.somers@biblibre.com> --- Pushed to master for 22.05, thanks to everybody involved 🦄 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to master |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Bug 27783 depends on bug 30345, which changed state. Bug 30345 Summary: Koha::BackgroundJob->enqueue should set borrowernumber=undef if no userenv https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30345 What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to master |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #59 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- *** Bug 30350 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |30656 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30656 [Bug 30656] Introduce worker_manager daemon -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |release-notes-needed -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Andrew Nugged <nugged@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nugged@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |30889 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30889 [Bug 30889] Background jobs lead to wrong/missing info in logs -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 wajasu <matted-34813@mypacks.net> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=31124 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |32394 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32394 [Bug 32394] Long tasks queue is never used -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #60 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Why do we need the DB column background_jobs.queue exactly? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #61 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- (In reply to Jonathan Druart from comment #60)
Why do we need the DB column background_jobs.queue exactly?
... In case we are defaulting to the DB (no broker). -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |32465 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32465 [Bug 32465] koha-worker debian script missing 'queue' in help -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |m.de.rooy@rijksmuseum.nl --- Comment #62 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Looking back at these patches, I am wondering if the misc script can support multiple queues in one process but koha-worker does not ? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #63 from Tomás Cohen Arazi <tomascohen@gmail.com> --- (In reply to Marcel de Rooy from comment #62)
Looking back at these patches, I am wondering if the misc script can support multiple queues in one process but koha-worker does not ?
True. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #64 from David Cook <dcook@prosentient.com.au> --- (In reply to Marcel de Rooy from comment #62)
Looking back at these patches, I am wondering if the misc script can support multiple queues in one process but koha-worker does not ?
koha-worker is just a service wrapper around background_jobs_worker.pl. koha-worker could be improved to support more permutations. None of my Koha instances are heavy users of background jobs so it hasn't been an issue for me, but I think Bywater are running into issues with not having enough workers available. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #65 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to David Cook from comment #64)
koha-worker is just a service wrapper around background_jobs_worker.pl. koha-worker could be improved to support more permutations. In QA I would probably have insisted on keeping things consistent. If you add it in the script, add it in the wrapper too.
None of my Koha instances are heavy users of background jobs so it hasn't been an issue for me, but I think Bywater are running into issues with not having enough workers available. The development here allows you to start workers per queue. But you need enough cores too obviously. I notice that the parent waits for the child to finish; if you code that differently you can fork simultaneously (if cores allow; could be parametrized) reducing the need for the queue differentiation again.
-- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #66 from Tomás Cohen Arazi <tomascohen@gmail.com> --- (In reply to Marcel de Rooy from comment #65)
(In reply to David Cook from comment #64)
koha-worker is just a service wrapper around background_jobs_worker.pl. koha-worker could be improved to support more permutations. In QA I would probably have insisted on keeping things consistent. If you add it in the script, add it in the wrapper too.
The idea was to have a mvp that allowed us to move to this workers from the CGI forking vestiges. And my reasoning around koha-worker was that we should have something that worked out of the box (i.e. koha-common.init launching things). While I like the idea of making koha-worker run more than one queue per process, my perception was that sysadmins would rather dockerize the background_job_worker.pl run to split the load accross servers.
None of my Koha instances are heavy users of background jobs so it hasn't been an issue for me, but I think Bywater are running into issues with not having enough workers available. The development here allows you to start workers per queue. But you need enough cores too obviously. I notice that the parent waits for the child to finish; if you code that differently you can fork simultaneously (if cores allow; could be parametrized) reducing the need for the queue differentiation again.
Yeah, we worked around what was already in core by the time this was added. There are bugs filled for: - Setting a max child count (for parallel processing within a queue) - ACKing on receiving the message -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 --- Comment #67 from David Cook <dcook@prosentient.com.au> --- (In reply to Tomás Cohen Arazi from comment #66)
Yeah, we worked around what was already in core by the time this was added. There are bugs filled for: - Setting a max child count (for parallel processing within a queue) - ACKing on receiving the message
Yeah we probably don't need to continue this conversation here. See Bug 32573 and Bug 32558 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |33334 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33334 [Bug 33334] koha-worker.service need update for background job queues -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|release-notes-needed | -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org