https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Lisette Scheer <lisette.scheer@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |tomascohen@gmail.com |ity.org | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Chip Halvorsen <Chip.Halvorsen@WestlakeLibrary.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |Chip.Halvorsen@WestlakeLibr | |ary.org -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au, | |tomascohen@gmail.com --- Comment #1 from Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> --- Would it be acceptable to make this feature depend on manually installing a RabbitMQ plugin? https://github.com/rabbitmq/rabbitmq-delayed-message-exchange/releases I don't know how to solve 'the STOMP use case' otherwise. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #2 from David Cook <dcook@prosentient.com.au> --- As noted on that RabbitMQ plugin link, having a task scheduler could solve the problem for both broker methods. Upon failure, a task could be enqueued to run X seconds later. That task could be to restart the background job. It could reset the job status and then for RabbitMQ mode it could send a new message. The number of background job tries could be tracked in the job. We need a task scheduler at some point anyway. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #3 from David Cook <dcook@prosentient.com.au> --- That said... what's our proposed use case? What's the background job that would benefit from an automatic retry? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #5 from David Cook <dcook@prosentient.com.au> --- (In reply to Tomás Cohen Arazi (tcohen) from comment #4)

- Plugin hook scheduling API calls to external API that could fail (they do fail, not often but do) and I want the job to be able to detect certain errors and schedule a retry.

To me, it sounds like this would benefit from a task scheduler, and then the retry logic is just part of the plugin.

- ES overwhelmed somehow, I would like the ES indexing job to be retried under certain circumstances.

That could probably be useful. I suppose many things where there is inter-process communication it can be handy to be able to retry up to X times. So long as there's some coding to make sure there's no race conditions. ES indexing should be fine since it should just be passing an ID rather than any stale data. -- Yeah overall I think a task scheduler solves this. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=41506 -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |42643 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42643 [Bug 42643] [OMNIBUS] Assorted performance and stability work -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |matt.blenkinsop@openfifth.c | |o.uk -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=24438 -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #6 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 200995 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200995&action=edit Bug 36702: Update database Patch from commit 5f132f9 -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #8 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 200997 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200997&action=edit Bug 36702: Add ability to retry failed background jobs with a max tries parameter Currently when a background job fails it just stops with a status of 'failed' and there is no way to have it run again. This is a problem for jobs that fail for transient reasons, like an Elasticsearch index update when the server is briefly overwhelmed, or a plugin job that calls an external API that is temporarily unavailable. This patch adds the ability to retry a failed job up to a maximum number of times. When a job throws an error the worker leaves the failed job alone so its messages and report are kept as history, and enqueues a new job that retries it. The new job points back at the one it is retrying through previous_job_id, so the whole chain of attempts can be followed. The maximum number of retries comes from, in order: * an explicit max_retries passed to enqueue() * the job type's default_max_retries() * BackgroundJobsDefaultMaxRetries system preference ( default 3 ). A value of 0 disables retries. Jobs that aren't safe to re-run with their original arguments ( batch modifications, imports, e-holdings creation, SUSHI harvesting and statistics pseudonymization ) override default_max_retries to 0 so they opt out. Retries don't all fire at once. The first retry runs immediately and each following retry waits an extra BackgroundJobsRetryDelay seconds ( default 30 ), tracked through the not_before column. The worker won't process a job before its not_before time, requeuing it in RabbitMQ mode or skipping it until the next poll in database mode, so retries work the same whether or not RabbitMQ is used. Test Plan: 1) Apply all the patches 2) Run updatedatabase.pl 3) Restart all the things! 4) Note the new columns on the background_jobs table ( max_retries, retries, previous_job_id, not_before ) and the two new system preferences, BackgroundJobsDefaultMaxRetries and BackgroundJobsRetryDelay, under Administration > System preferences > Administration > Jobs! 5) Leave BackgroundJobsDefaultMaxRetries at 3 and BackgroundJobsRetryDelay at 30 6) Make sure JobsNotificationMethod is set to 'STOMP' 7) Restart background jobs workers 8) Enqueue a job that will fail. In the Koha shell run: perl -e 'use Koha::BackgroundJob::TestTransport; Koha::BackgroundJob::TestTransport->new->enqueue( { transport_id => 999999 } );' 9) Browse to Administration > Manage jobs and refresh as the job runs 10) Note the original job ends as 'failed' and a new job is enqueued that links back to it through 'Retry of' 11) Note the first retry runs immediately ( Retries 1 / 3 ), a second is held back about 30 seconds ( Retries 2 / 3 ) and a third about 60 seconds ( Retries 3 / 3 ) 12) Note that once retries reaches BackgroundJobsDefaultMaxRetries ( 3 ) no further retry is made 13) Set JobsNotificationMethod to 'polling' 14) Restart background jobs workers again 15) Repeat step 8 to enqueue another failing job 16) Note similar retry chain is built -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #10 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 200999 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200999&action=edit Bug 36702: Show background job retry information in the staff interface Patch from commit 81ef67e -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #12 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 201001 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201001&action=edit Bug 36702: Add unit tests Patch from commit ad6f4da -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |24438 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24438 [Bug 24438] index can fail on timeout -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |ASSIGNED --- Comment #14 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Still needs follow-up for elastic indexer. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #201002|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #16 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 201063 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201063&action=edit Bug 36702: Split oversized Elasticsearch index batches instead of retrying them Patch from commit cbeec60 -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |24438 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24438 [Bug 24438] index can fail on timeout -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Clemens Tubach <clemens.tubach@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #200995|0 |1 is obsolete| | Attachment #200996|0 |1 is obsolete| | Attachment #200997|0 |1 is obsolete| | Attachment #200998|0 |1 is obsolete| | Attachment #200999|0 |1 is obsolete| | Attachment #201000|0 |1 is obsolete| | Attachment #201001|0 |1 is obsolete| | Attachment #201062|0 |1 is obsolete| | Attachment #201063|0 |1 is obsolete| | Attachment #201064|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #19 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 201478 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201478&action=edit Bug 36702: Move background job broker publishing into a reusable method Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #21 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 201480 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201480&action=edit Bug 36702: Add retry fields to the background jobs REST API Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #23 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 201482 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201482&action=edit Bug 36702: Don't automatically retry background jobs that aren't safe to re-run Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #25 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 201484 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201484&action=edit Bug 36702: Retry failed Elasticsearch index jobs Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 --- Comment #27 from Clemens Tubach <clemens.tubach@kit.edu> --- Created attachment 201486 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201486&action=edit Bug 36702: Update Schema files [DO NOT PUSH] Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |andrew@bywatersolutions.com Blocks| |43016 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43016 [Bug 43016] [OMNIBUS] Server resource protection -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Patch doesn't apply -- You are receiving this mail because: You are watching all bug changes.