[Bug 36702] New: Background Jobs:Add option to retry failed jobs with a max tries parameter
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Bug ID: 36702 Summary: Background Jobs:Add option to retry failed jobs with a max tries parameter Change sponsored?: --- Product: Koha Version: unspecified Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Architecture, internals, and plumbing Assignee: koha-bugs@lists.koha-community.org Reporter: lisette.scheer@bywatersolutions.com QA Contact: testopia@bugs.koha-community.org Currently the background jobs just fail. It would be helpful if there was a path to add or set background jobs with a parameter to retry x times (probably with a y seconds interval between?) especially now that plugins can add background jobs. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Lisette Scheer <lisette.scheer@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Background Jobs:Add option |Background Jobs: Add option |to retry failed jobs with a |to retry failed jobs with a |max tries parameter |max tries parameter -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Lisette Scheer <lisette.scheer@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |tomascohen@gmail.com |ity.org | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=35842 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Chip Halvorsen <Chip.Halvorsen@WestlakeLibrary.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |Chip.Halvorsen@WestlakeLibr | |ary.org -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Katie Bliss <kebliss@dmpl.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |kebliss@dmpl.org -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au, | |tomascohen@gmail.com --- Comment #1 from Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> --- Would it be acceptable to make this feature depend on manually installing a RabbitMQ plugin? https://github.com/rabbitmq/rabbitmq-delayed-message-exchange/releases I don't know how to solve 'the STOMP use case' otherwise. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #2 from David Cook <dcook@prosentient.com.au> --- As noted on that RabbitMQ plugin link, having a task scheduler could solve the problem for both broker methods. Upon failure, a task could be enqueued to run X seconds later. That task could be to restart the background job. It could reset the job status and then for RabbitMQ mode it could send a new message. The number of background job tries could be tracked in the job. We need a task scheduler at some point anyway. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=39605 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #3 from David Cook <dcook@prosentient.com.au> --- That said... what's our proposed use case? What's the background job that would benefit from an automatic retry? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #4 from Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> --- (In reply to David Cook from comment #3)
That said... what's our proposed use case? What's the background job that would benefit from an automatic retry?
I have some real life use cases - Plugin hook scheduling API calls to external API that could fail (they do fail, not often but do) and I want the job to be able to detect certain errors and schedule a retry. - ES overwhelmed somehow, I would like the ES indexing job to be retried under certain circumstances. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #5 from David Cook <dcook@prosentient.com.au> --- (In reply to Tomás Cohen Arazi (tcohen) from comment #4)
- Plugin hook scheduling API calls to external API that could fail (they do fail, not often but do) and I want the job to be able to detect certain errors and schedule a retry.
To me, it sounds like this would benefit from a task scheduler, and then the retry logic is just part of the plugin.
- ES overwhelmed somehow, I would like the ES indexing job to be retried under certain circumstances.
That could probably be useful. I suppose many things where there is inter-process communication it can be handy to be able to retry up to X times. So long as there's some coding to make sure there's no race conditions. ES indexing should be fine since it should just be passing an ID rather than any stale data. -- Yeah overall I think a task scheduler solves this. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |35092 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35092 [Bug 35092] [OMNIBUS] Remaining background job/worker issues -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=41506 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |clemens.tubach@kit.edu, | |martin.renvoize@openfifth.c | |o.uk, | |michaela.sieber@kit.edu, | |raphael.straub@kit.edu -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |42643 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42643 [Bug 42643] [OMNIBUS] Assorted performance and stability work -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|tomascohen@gmail.com |kyle@bywatersolutions.com CC| |kyle@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |matt.blenkinsop@openfifth.c | |o.uk -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Initiative type|--- |Feature Target Milestone|--- |26.11 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=24438 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #6 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 200995 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200995&action=edit Bug 36702: Update database Patch from commit 5f132f9 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #7 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 200996 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200996&action=edit Bug 36702: Move background job broker publishing into a reusable method Patch from commit 35919ab -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #8 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 200997 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200997&action=edit Bug 36702: Add ability to retry failed background jobs with a max tries parameter Currently when a background job fails it just stops with a status of 'failed' and there is no way to have it run again. This is a problem for jobs that fail for transient reasons, like an Elasticsearch index update when the server is briefly overwhelmed, or a plugin job that calls an external API that is temporarily unavailable. This patch adds the ability to retry a failed job up to a maximum number of times. When a job throws an error the worker leaves the failed job alone so its messages and report are kept as history, and enqueues a new job that retries it. The new job points back at the one it is retrying through previous_job_id, so the whole chain of attempts can be followed. The maximum number of retries comes from, in order: * an explicit max_retries passed to enqueue() * the job type's default_max_retries() * BackgroundJobsDefaultMaxRetries system preference ( default 3 ). A value of 0 disables retries. Jobs that aren't safe to re-run with their original arguments ( batch modifications, imports, e-holdings creation, SUSHI harvesting and statistics pseudonymization ) override default_max_retries to 0 so they opt out. Retries don't all fire at once. The first retry runs immediately and each following retry waits an extra BackgroundJobsRetryDelay seconds ( default 30 ), tracked through the not_before column. The worker won't process a job before its not_before time, requeuing it in RabbitMQ mode or skipping it until the next poll in database mode, so retries work the same whether or not RabbitMQ is used. Test Plan: 1) Apply all the patches 2) Run updatedatabase.pl 3) Restart all the things! 4) Note the new columns on the background_jobs table ( max_retries, retries, previous_job_id, not_before ) and the two new system preferences, BackgroundJobsDefaultMaxRetries and BackgroundJobsRetryDelay, under Administration > System preferences > Administration > Jobs! 5) Leave BackgroundJobsDefaultMaxRetries at 3 and BackgroundJobsRetryDelay at 30 6) Make sure JobsNotificationMethod is set to 'STOMP' 7) Restart background jobs workers 8) Enqueue a job that will fail. In the Koha shell run: perl -e 'use Koha::BackgroundJob::TestTransport; Koha::BackgroundJob::TestTransport->new->enqueue( { transport_id => 999999 } );' 9) Browse to Administration > Manage jobs and refresh as the job runs 10) Note the original job ends as 'failed' and a new job is enqueued that links back to it through 'Retry of' 11) Note the first retry runs immediately ( Retries 1 / 3 ), a second is held back about 30 seconds ( Retries 2 / 3 ) and a third about 60 seconds ( Retries 3 / 3 ) 12) Note that once retries reaches BackgroundJobsDefaultMaxRetries ( 3 ) no further retry is made 13) Set JobsNotificationMethod to 'polling' 14) Restart background jobs workers again 15) Repeat step 8 to enqueue another failing job 16) Note similar retry chain is built -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #9 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 200998 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200998&action=edit Bug 36702: Add retry fields to the background jobs REST API Patch from commit 79e7771 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #10 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 200999 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200999&action=edit Bug 36702: Show background job retry information in the staff interface Patch from commit 81ef67e -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #11 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 201000 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201000&action=edit Bug 36702: Don't automatically retry background jobs that aren't safe to re-run Patch from commit 960184f -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #12 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 201001 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201001&action=edit Bug 36702: Add unit tests Patch from commit ad6f4da -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #13 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 201002 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201002&action=edit Bug 36702: Update Schema files [DO NOT PUSH] Patch from commit 7d789ab -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |24438 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24438 [Bug 24438] index can fail on timeout -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks|24438 | Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24438 [Bug 24438] index can fail on timeout -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |ASSIGNED --- Comment #14 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Still needs follow-up for elastic indexer. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #201002|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #15 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 201062 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201062&action=edit Bug 36702: Retry failed Elasticsearch index jobs Patch from commit 5d000f3 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #16 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 201063 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201063&action=edit Bug 36702: Split oversized Elasticsearch index batches instead of retrying them Patch from commit cbeec60 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #17 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 201064 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201064&action=edit Bug 36702: Update Schema files [DO NOT PUSH] Patch from commit 4c59ed3 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |24438 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24438 [Bug 24438] index can fail on timeout -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Clemens Tubach <clemens.tubach@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Clemens Tubach <clemens.tubach@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #200995|0 |1 is obsolete| | Attachment #200996|0 |1 is obsolete| | Attachment #200997|0 |1 is obsolete| | Attachment #200998|0 |1 is obsolete| | Attachment #200999|0 |1 is obsolete| | Attachment #201000|0 |1 is obsolete| | Attachment #201001|0 |1 is obsolete| | Attachment #201062|0 |1 is obsolete| | Attachment #201063|0 |1 is obsolete| | Attachment #201064|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #18 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 201477 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201477&action=edit Bug 36702: Update database Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #19 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 201478 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201478&action=edit Bug 36702: Move background job broker publishing into a reusable method Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #20 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 201479 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201479&action=edit Bug 36702: Add ability to retry failed background jobs with a max tries parameter Currently when a background job fails it just stops with a status of 'failed' and there is no way to have it run again. This is a problem for jobs that fail for transient reasons, like an Elasticsearch index update when the server is briefly overwhelmed, or a plugin job that calls an external API that is temporarily unavailable. This patch adds the ability to retry a failed job up to a maximum number of times. When a job throws an error the worker leaves the failed job alone so its messages and report are kept as history, and enqueues a new job that retries it. The new job points back at the one it is retrying through previous_job_id, so the whole chain of attempts can be followed. The maximum number of retries comes from, in order: * an explicit max_retries passed to enqueue() * the job type's default_max_retries() * BackgroundJobsDefaultMaxRetries system preference ( default 3 ). A value of 0 disables retries. Jobs that aren't safe to re-run with their original arguments ( batch modifications, imports, e-holdings creation, SUSHI harvesting and statistics pseudonymization ) override default_max_retries to 0 so they opt out. Retries don't all fire at once. The first retry runs immediately and each following retry waits an extra BackgroundJobsRetryDelay seconds ( default 30 ), tracked through the not_before column. The worker won't process a job before its not_before time, requeuing it in RabbitMQ mode or skipping it until the next poll in database mode, so retries work the same whether or not RabbitMQ is used. Test Plan: 1) Apply all the patches 2) Run updatedatabase.pl 3) Restart all the things! 4) Note the new columns on the background_jobs table ( max_retries, retries, previous_job_id, not_before ) and the two new system preferences, BackgroundJobsDefaultMaxRetries and BackgroundJobsRetryDelay, under Administration > System preferences > Administration > Jobs! 5) Leave BackgroundJobsDefaultMaxRetries at 3 and BackgroundJobsRetryDelay at 30 6) Make sure JobsNotificationMethod is set to 'STOMP' 7) Restart background jobs workers 8) Enqueue a job that will fail. In the Koha shell run: perl -e 'use Koha::BackgroundJob::TestTransport; Koha::BackgroundJob::TestTransport->new->enqueue( { transport_id => 999999 } );' 9) Browse to Administration > Manage jobs and refresh as the job runs 10) Note the original job ends as 'failed' and a new job is enqueued that links back to it through 'Retry of' 11) Note the first retry runs immediately ( Retries 1 / 3 ), a second is held back about 30 seconds ( Retries 2 / 3 ) and a third about 60 seconds ( Retries 3 / 3 ) 12) Note that once retries reaches BackgroundJobsDefaultMaxRetries ( 3 ) no further retry is made 13) Set JobsNotificationMethod to 'polling' 14) Restart background jobs workers again 15) Repeat step 8 to enqueue another failing job 16) Note similar retry chain is built Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #21 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 201480 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201480&action=edit Bug 36702: Add retry fields to the background jobs REST API Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #22 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 201481 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201481&action=edit Bug 36702: Show background job retry information in the staff interface Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #23 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 201482 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201482&action=edit Bug 36702: Don't automatically retry background jobs that aren't safe to re-run Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #24 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 201483 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201483&action=edit Bug 36702: Add unit tests Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #25 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 201484 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201484&action=edit Bug 36702: Retry failed Elasticsearch index jobs Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #26 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 201485 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201485&action=edit Bug 36702: Split oversized Elasticsearch index batches instead of retrying them Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #27 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 201486 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201486&action=edit Bug 36702: Update Schema files [DO NOT PUSH] Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #28 from Michaela Sieber <michaela.sieber@kit.edu> --- Thanks Kyle! We were a bit surprised because we expected the time interval between retries to remain constant as defined in syspref BackgroundJobsRetryDelay. However, that is not the case—the time gap between successive retries keeps increasing. For example, with 5 retries and an interval of 10, the timing looks like this: +----+--------+---------------------+---------------------+----------+-------------+---------+-----------------+---------------------+ | id | status | enqueued_on | started_on | ended_on | max_retries | retries | previous_job_id | not_before | +----+--------+---------------------+---------------------+----------+-------------+---------+-----------------+---------------------+ | 12 | failed | 2026-07-03 09:39:28 | 2026-07-03 09:39:34 | NULL | 5 | 0 | NULL | NULL | | 13 | failed | 2026-07-03 09:39:34 | 2026-07-03 09:39:35 | NULL | 5 | 1 | 12 | 2026-07-03 09:39:34 | -> 1. retry: 0 secs | 14 | failed | 2026-07-03 09:39:35 | 2026-07-03 09:39:45 | NULL | 5 | 2 | 13 | 2026-07-03 09:39:45 | -> 2. retry: 10 secs | 15 | failed | 2026-07-03 09:39:45 | 2026-07-03 09:40:05 | NULL | 5 | 3 | 14 | 2026-07-03 09:40:05 | -> 3. retry: 20 secs | 16 | failed | 2026-07-03 09:40:06 | 2026-07-03 09:40:36 | NULL | 5 | 4 | 15 | 2026-07-03 09:40:36 | -> 4. retry: 30 secs | 17 | failed | 2026-07-03 09:40:36 | 2026-07-03 09:41:17 | NULL | 5 | 5 | 16 | 2026-07-03 09:41:16 | -> 5. retry: 40 secs In addition to the test plan, we also verified whether ERM havesting jobs are retried. This was not the case, as described in comment 8 of the test plan. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |andrew@bywatersolutions.com Blocks| |43016 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43016 [Bug 43016] [OMNIBUS] Server resource protection -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Initiative type|Feature |--- -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Patch doesn't apply -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #29 from Andrew Fuerste-Henry <andrew@bywatersolutions.com> --- Bug 42669 changed some things here that I don't know how to reconcile. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|kyle@bywatersolutions.com |tomascohen@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |zoho.roboto@bywatersolution | |s.com -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org