[Bug 31629] New: Cleanup database needs a flag for removing elastic index entries from background_jobs table
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31629 Bug ID: 31629 Summary: Cleanup database needs a flag for removing elastic index entries from background_jobs table Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: System Administration Assignee: koha-bugs@lists.koha-community.org Reporter: wizzyrea@gmail.com QA Contact: testopia@bugs.koha-community.org CC: gmcharlt@gmail.com Bug 27344 introduced using the background worker to handle elastic indexing, which means the background_jobs table can end up with a lot of entries for elastic search updates. Currently these are not cleaned up, we either need a job in cleanup_database cron to get rid of these, or they should be removed after processing. If we have a cleanup database job, the default interval for deletions should be 1 day. Cheers, Liz -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31629 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31629 --- Comment #1 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 141837 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=141837&action=edit Bug 31629: Add elasticqueue options to cleanup_database This patch adds an elasticqueue option to cleanup_database.pl to allow for purging completed reindexing jobs To test: 1 - Enable elastic search in Koha 2 - perl misc/maintenance/touch_all_items.pl 3 - Check db and note there are a bunch of elastic reindex jobs 4 - Update to make them old UPDATE background_jobs SET ended_on = '2022-10-01 00:00:00' WHERE type = 'update_elastic_index' 5 - perl misc/cronjobs/cleanup_database.pl 6 - Note elasticqueue entry shows in help 7 - perl misc/cronjobs/cleanup_database.pl --elasticqueue 1 -v 8 - Note that elasticqueue would have been cleared 9 - perl misc/cronjobs/cleanup_database.pl --elasticqueue 1 -v --confirm 10 - Note that number of entries deleted is reported 11 - Confirm in staff interface that jobs are gone: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl (Uncheck 'Current jobs only') -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31629 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31629 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #141837|0 |1 is obsolete| | --- Comment #2 from David Nind <david@davidnind.com> --- Created attachment 141850 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=141850&action=edit Bug 31629: Add elasticqueue options to cleanup_database This patch adds an elasticqueue option to cleanup_database.pl to allow for purging completed reindexing jobs To test: 1 - Enable elastic search in Koha 2 - perl misc/maintenance/touch_all_items.pl 3 - Check db and note there are a bunch of elastic reindex jobs 4 - Update to make them old UPDATE background_jobs SET ended_on = '2022-10-01 00:00:00' WHERE type = 'update_elastic_index' 5 - perl misc/cronjobs/cleanup_database.pl 6 - Note elasticqueue entry shows in help 7 - perl misc/cronjobs/cleanup_database.pl --elasticqueue 1 -v 8 - Note that elasticqueue would have been cleared 9 - perl misc/cronjobs/cleanup_database.pl --elasticqueue 1 -v --confirm 10 - Note that number of entries deleted is reported 11 - Confirm in staff interface that jobs are gone: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl (Uncheck 'Current jobs only') Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31629 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |nick@bywatersolutions.com |ity.org | --- Comment #3 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Hm... a week ago - I think it's cookie time! ;) -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31629 --- Comment #4 from Frédéric Demians <frederic@tamil.fr> --- Created attachment 142492 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=142492&action=edit Bug 31154: Avoid unecessary loop We don't need to loop over the subfield and enter this block if there is no record. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31629 Frédéric Demians <frederic@tamil.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |frederic@tamil.fr Attachment #142492|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31629 --- Comment #5 from Frédéric Demians <frederic@tamil.fr> --- Above, it was a false attachement, made by mistake. Sorry. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31629 Frédéric Demians <frederic@tamil.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Needs Signoff --- Comment #6 from Frédéric Demians <frederic@tamil.fr> --- The default value for deletion is set to 30 in DEFAULT_ELASTICQ_PURGEDAYS. Not 1. Is it intended? What do you think about another option for purging all other finished jobs? Something like --jobqueue. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31629 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off --- Comment #7 from Nick Clemens <nick@bywatersolutions.com> --- (In reply to Frédéric Demians from comment #6)
The default value for deletion is set to 30 in DEFAULT_ELASTICQ_PURGEDAYS. Not 1. Is it intended?
Matching the default for zebraqueue
What do you think about another option for purging all other finished jobs? Something like --jobqueue.
I think it is a great idea, please file a follow-up bug -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31629 Björn Nylén <bjorn.nylen@ub.lu.se> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bjorn.nylen@ub.lu.se --- Comment #8 from Björn Nylén <bjorn.nylen@ub.lu.se> --- I made a more generalized addition to cleanup_database.pl before I saw this. (moved to 22.05 last week so wanted this quickly). Can add it as a suggestion if people are interested? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31629 Frédéric Demians <frederic@tamil.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #141850|0 |1 is obsolete| | --- Comment #9 from Frédéric Demians <frederic@tamil.fr> --- Created attachment 142501 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=142501&action=edit Bug 31629: Add elasticqueue options to cleanup_database This patch adds an elasticqueue option to cleanup_database.pl to allow for purging completed reindexing jobs To test: 1 - Enable elastic search in Koha 2 - perl misc/maintenance/touch_all_items.pl 3 - Check db and note there are a bunch of elastic reindex jobs 4 - Update to make them old UPDATE background_jobs SET ended_on = '2022-10-01 00:00:00' WHERE type = 'update_elastic_index' 5 - perl misc/cronjobs/cleanup_database.pl 6 - Note elasticqueue entry shows in help 7 - perl misc/cronjobs/cleanup_database.pl --elasticqueue 1 -v 8 - Note that elasticqueue would have been cleared 9 - perl misc/cronjobs/cleanup_database.pl --elasticqueue 1 -v --confirm 10 - Note that number of entries deleted is reported 11 - Confirm in staff interface that jobs are gone: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl (Uncheck 'Current jobs only') Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Frédéric Demians <f.demians@tamil.fr> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31629 --- Comment #10 from Frédéric Demians <frederic@tamil.fr> --- Created attachment 142502 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=142502&action=edit Bug 31629: Add jobsqueue options to cleanup_database This patch adds an jobsqueue option to cleanup_database.pl to allow for purging background jobs queue There is also a fix for --elasticqueue default value To test: 1. On a DB with a populated background_jobs table, identify in this table entries older than X days. 2. perl misc/cronjobs/cleanup_database.pl Note elasticqueue entry shows in help 3. perl misc/cronjobs/cleanup_database.pl --elasticqueue -v Note this message: Elastic queue (background jobs) purge triggered for 30 days 4. perl misc/cronjobs/cleanup_database.pl --jobsqueue -v Note this message: Background jobs queue purge triggered for 30 days. 5. perl misc/cronjobs/cleanup_database.pl --jobsqueue 1 -v Background jobs queue purge triggered for 1 day. 6. perl misc/cronjobs/cleanup_database.pl --jobsqueue 1 -v --confirm Note that number of entries deleted is reporte 7. In Koha > Admin > Background jobs, see the purge queue. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31629 Frédéric Demians <frederic@tamil.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31629 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31629 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #142501|0 |1 is obsolete| | --- Comment #11 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 142508 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=142508&action=edit Bug 31629: Add elasticqueue options to cleanup_database This patch adds an elasticqueue option to cleanup_database.pl to allow for purging completed reindexing jobs To test: 1 - Enable elastic search in Koha 2 - perl misc/maintenance/touch_all_items.pl 3 - Check db and note there are a bunch of elastic reindex jobs 4 - Update to make them old UPDATE background_jobs SET ended_on = '2022-10-01 00:00:00' WHERE type = 'update_elastic_index' 5 - perl misc/cronjobs/cleanup_database.pl 6 - Note elasticqueue entry shows in help 7 - perl misc/cronjobs/cleanup_database.pl --elasticqueue 1 -v 8 - Note that elasticqueue would have been cleared 9 - perl misc/cronjobs/cleanup_database.pl --elasticqueue 1 -v --confirm 10 - Note that number of entries deleted is reported 11 - Confirm in staff interface that jobs are gone: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl (Uncheck 'Current jobs only') Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Frédéric Demians <f.demians@tamil.fr> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31629 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #142502|0 |1 is obsolete| | --- Comment #12 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 142509 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=142509&action=edit Bug 31629: Add jobsqueue options to cleanup_database This patch adds an jobsqueue option to cleanup_database.pl to allow for purging background jobs queue There is also a fix for --elasticqueue default value To test: 1. On a DB with a populated background_jobs table, identify in this table entries older than X days. 2. perl misc/cronjobs/cleanup_database.pl Note elasticqueue entry shows in help 3. perl misc/cronjobs/cleanup_database.pl --elasticqueue -v Note this message: Elastic queue (background jobs) purge triggered for 30 days 4. perl misc/cronjobs/cleanup_database.pl --jobsqueue -v Note this message: Background jobs queue purge triggered for 30 days. 5. perl misc/cronjobs/cleanup_database.pl --jobsqueue 1 -v Background jobs queue purge triggered for 1 day. 6. perl misc/cronjobs/cleanup_database.pl --jobsqueue 1 -v --confirm Note that number of entries deleted is reporte 7. In Koha > Admin > Background jobs, see the purge queue. Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31629 --- Comment #13 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 142510 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=142510&action=edit Bug 31629: (follow-up) Fix spelling of background Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31629 --- Comment #14 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Björn Nylén from comment #8)
I made a more generalized addition to cleanup_database.pl before I saw this. (moved to 22.05 last week so wanted this quickly). Can add it as a suggestion if people are interested?
Hi Björn, do you mean for tasks/jobs in general? A separate bug as mentioned in comment#7 would be great. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31629 --- Comment #15 from Björn Nylén <bjorn.nylen@ub.lu.se> --- (In reply to Katrin Fischer from comment #14)
(In reply to Björn Nylén from comment #8)
I made a more generalized addition to cleanup_database.pl before I saw this. (moved to 22.05 last week so wanted this quickly). Can add it as a suggestion if people are interested?
Hi Björn, do you mean for tasks/jobs in general? A separate bug as mentioned in comment#7 would be great.
Yeah, I made it so you could specify what jobtypes to purge and when. Obv mostly needed for elastic jobs but eventually you'll want to get rid of the other types. Will post a bug in the morning. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31629 Björn Nylén <bjorn.nylen@ub.lu.se> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=31969 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31629 --- Comment #16 from Frédéric Demians <frederic@tamil.fr> --- In my opinion bug 31969 should be privileged to this one. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31629 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |RESOLVED Resolution|--- |DUPLICATE --- Comment #17 from Nick Clemens <nick@bywatersolutions.com> --- *** This bug has been marked as a duplicate of bug 31969 *** -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org