[Bug 35948] New: cleanup_database.pl should remove not only finished background jobs
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35948 Bug ID: 35948 Summary: cleanup_database.pl should remove not only finished background jobs Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Command-line Utilities Assignee: koha-bugs@lists.koha-community.org Reporter: fridolin.somers@biblibre.com QA Contact: testopia@bugs.koha-community.org CC: robin@catalyst.net.nz Bug 31969 changed cleanup_database.pl so it removes only finished background jobs. But old background jobs with other statuses will never be removed. I see no use to that -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35948 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED Assignee|koha-bugs@lists.koha-commun |fridolin.somers@biblibre.co |ity.org |m Depends on| |31969 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31969 [Bug 31969] Options for cleanup_database.pl to remove finished jobs from the background job queue -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35948 --- Comment #1 from Fridolin Somers <fridolin.somers@biblibre.com> --- So we should use "enqueued_on" instead of "ended_on" and no filter on status -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35948 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|--- |Trivial patch Status|ASSIGNED |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35948 --- Comment #2 from Fridolin Somers <fridolin.somers@biblibre.com> --- Created attachment 161646 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=161646&action=edit Bug 35948: cleanup_database.pl should remove not only finished background jobs Bug 31969 changed cleanup_database.pl so it removes only finished background jobs. But old background jobs with other statuses will never be removed. So we should use "enqueued_on" instead of "ended_on" and no filter on status Test plan : 1) Create a background job with for example a batch item modification 2) Simulate ancien job via SQL : update background_jobs set status='failed',enqueued_on=DATE_SUB(NOW(),INTERVAL 10 DAY) where id=X 3) Run misc/cronjobs/cleanup_database.pl --jobs-days 9 --jobs-type all -v --confirm 4) Check the job is removed -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35948 --- Comment #3 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Currently we have a lot of problems with jobs not finishing. Would this not only hide the problem? I feel it should be an option and not a change of behaviour. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35948 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |victor@tuxayo.net --- Comment #4 from Victor Grousset/tuxayo <victor@tuxayo.net> --- @Fridolin: see bug 35092 What are the other "final" statues that should be cleaned? Certainly there should be "cancelled". And on the long run, "failed" also. Not sure about now because that might hide the current issues if --jobs-days is too small. But maybe it ok, I don't know. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35948 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |In Discussion --- Comment #5 from Fridolin Somers <fridolin.somers@biblibre.com> --- Ah sure, cancelled is missing. I set status to needs discussion ;) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35948 --- Comment #6 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Anything else to cleanup besides cancelled and failed? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35948 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|fridolin.somers@biblibre.co |nick@bywatersolutions.com |m | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35948 --- Comment #7 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 199476 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199476&action=edit Bug 35948: Add jobs_unfinished_days and jobs_unfinished_type to cleanup_database.pl This patch adds two new parameters to control the deletion of background jobs that did not complete. Some libraries may want to keep a failed job longer than a completed one for debugging reasons. To test: 0 - Apply patch 1 - Make some old failed jobs: INSERT INTO background_jobs (status,type,enqueued_on) VALUES ('started','update_elastic_index','2020-01-01'), ('failed','update_elastic_index','2026-01-01'), ('started','other_type','2021-01-01'), ('failed','other_type','2026-01-01'); 2 - Confirm defaults of 365 and update_elastic_index for new params: perl misc/cronjobs/cleanup_database.pl -v --jobs-unfinished-days 3 - Should indicate 1 job older than 365 of type update_elastic_index would have been deleted 4 - Confirm days param works perl misc/cronjobs/cleanup_database.pl -v --jobs-unfinished-days 30 5 - Shoudl find 2 elastic jobs 6 - Confirm type param perl misc/cronjobs/cleanup_database.pl -v --jobs-unfinished-days --jobs-unfinished-type=other_type 7 - Should find 1 8 - Confirm type and days perl misc/cronjobs/cleanup_database.pl -v --jobs-unfinished-days 30 --jobs-unfinished-type=other_type 9 - Should find 2 10 - Confirm repeated type perl misc/cronjobs/cleanup_database.pl -v --jobs-unfinished-days=30 --jobs-unfinished-type=other_type --jobs-unfinished-type=update_elastic_index 11 - Finds 4 12 - Confirm deletion perl misc/cronjobs/cleanup_database.pl -v --jobs-unfinished-days=30 --jobs-unfinished-type=other_type --jobs-unfinished-type=update_elastic_index --confirm 13 - All your made up jobs are removed -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35948 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #161646|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35948 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Discussion |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35948 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35948 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #199476|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35948 --- Comment #8 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 199763 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199763&action=edit Bug 35948: Add jobs_unfinished_days and jobs_unfinished_type to cleanup_database.pl This patch adds two new parameters to control the deletion of background jobs that did not complete. Some libraries may want to keep a failed job longer than a completed one for debugging reasons. To test: 0 - Apply patch 1 - Make some old failed jobs: INSERT INTO background_jobs (status,type,enqueued_on) VALUES ('started','update_elastic_index','2020-01-01'), ('failed','update_elastic_index','2026-01-01'), ('started','other_type','2021-01-01'), ('failed','other_type','2026-01-01'); 2 - Confirm defaults of 365 and update_elastic_index for new params: perl misc/cronjobs/cleanup_database.pl -v --jobs-unfinished-days 3 - Should indicate 1 job older than 365 of type update_elastic_index would have been deleted 4 - Confirm days param works perl misc/cronjobs/cleanup_database.pl -v --jobs-unfinished-days 30 5 - Shoudl find 2 elastic jobs 6 - Confirm type param perl misc/cronjobs/cleanup_database.pl -v --jobs-unfinished-days --jobs-unfinished-type=other_type 7 - Should find 1 8 - Confirm type and days perl misc/cronjobs/cleanup_database.pl -v --jobs-unfinished-days 30 --jobs-unfinished-type=other_type 9 - Should find 2 10 - Confirm repeated type perl misc/cronjobs/cleanup_database.pl -v --jobs-unfinished-days=30 --jobs-unfinished-type=other_type --jobs-unfinished-type=update_elastic_index 11 - Finds 4 12 - Confirm deletion perl misc/cronjobs/cleanup_database.pl -v --jobs-unfinished-days=30 --jobs-unfinished-type=other_type --jobs-unfinished-type=update_elastic_index --confirm 13 - All your made up jobs are removed Signed-off-by: Trevor Diamond <trevor.diamond@mainlib.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35948 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lisette@bywatersolutions.co | |m QA Contact|testopia@bugs.koha-communit |martin.renvoize@openfifth.c |y.org |o.uk -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35948 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|Trivial patch |Small patch Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35948 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #199763|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35948 --- Comment #9 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 202001 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202001&action=edit Bug 35948: Add jobs_unfinished_days and jobs_unfinished_type to cleanup_database.pl This patch adds two new parameters to control the deletion of background jobs that did not complete. Some libraries may want to keep a failed job longer than a completed one for debugging reasons. To test: 0 - Apply patch 1 - Make some old failed jobs: INSERT INTO background_jobs (status,type,enqueued_on) VALUES ('started','update_elastic_index','2020-01-01'), ('failed','update_elastic_index','2026-01-01'), ('started','other_type','2021-01-01'), ('failed','other_type','2026-01-01'); 2 - Confirm defaults of 365 and update_elastic_index for new params: perl misc/cronjobs/cleanup_database.pl -v --jobs-unfinished-days 3 - Should indicate 1 job older than 365 of type update_elastic_index would have been deleted 4 - Confirm days param works perl misc/cronjobs/cleanup_database.pl -v --jobs-unfinished-days 30 5 - Shoudl find 2 elastic jobs 6 - Confirm type param perl misc/cronjobs/cleanup_database.pl -v --jobs-unfinished-days --jobs-unfinished-type=other_type 7 - Should find 1 8 - Confirm type and days perl misc/cronjobs/cleanup_database.pl -v --jobs-unfinished-days 30 --jobs-unfinished-type=other_type 9 - Should find 2 10 - Confirm repeated type perl misc/cronjobs/cleanup_database.pl -v --jobs-unfinished-days=30 --jobs-unfinished-type=other_type --jobs-unfinished-type=update_elastic_index 11 - Finds 4 12 - Confirm deletion perl misc/cronjobs/cleanup_database.pl -v --jobs-unfinished-days=30 --jobs-unfinished-type=other_type --jobs-unfinished-type=update_elastic_index --confirm 13 - All your made up jobs are removed Signed-off-by: Trevor Diamond <trevor.diamond@mainlib.org> Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35948 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |26.11.00 released in| | Status|Passed QA |Pushed to main -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35948 --- Comment #10 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Thanks everyone! Pushed to main for 26.11! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35948 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |clemens.tubach@kit.edu, | |katrin.fischer@bsz-bw.de, | |michaela.sieber@kit.edu -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35948 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC|katrin.fischer@bsz-bw.de | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35948 Lucas Gass (lukeg) <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to main |Pushed to stable Version(s)|26.11.00 |26.11.00,26.05.03 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35948 --- Comment #11 from Lucas Gass (lukeg) <lucas@bywatersolutions.com> --- Nice work everyone! Pushed to 26.05.x for the upcoming 26.05.03 release. -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org