[Bug 32250] New: link_bibs_to_authorities generates too many background jobs
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32250 Bug ID: 32250 Summary: link_bibs_to_authorities generates too many background jobs Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: MARC Authority data support Assignee: koha-bugs@lists.koha-community.org Reporter: nick@bywatersolutions.com QA Contact: testopia@bugs.koha-community.org When running the linker job we call ModBiblio for every linked record We should pass skip_record_index and skip_holds_queue and capture the biblionumbers - we can then generate a job per commit, rather than per record -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32250 --- Comment #1 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 145305 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145305&action=edit Bug 32250: Linking authorities does not change holdability If the only change is a linked heading, we don't need to rebuild the holds queue To test: 1 - Set preference RealTimeHoldsQueue to enable 2 - Run link_bibs_to_authorities 3 - Note holds queue jobs generated 4 - Apply patch 5 - repeat 6 - No holds queue updates -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32250 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32250 --- Comment #2 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 145319 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145319&action=edit Bug 32250: Only reindex records when commiting This patch adds an array to catch updated bibs, and defers indexing until a batch of changes is committed To test: 1 - Set LinkerModule system preference to either first or last match Alternate this between runs of the linker to ensure changes are made 2 - Set SearchEngine to Elasticsearch and reindex (to ensure index is built) 3 - perl misc/link_bibs_to_authorities.pl -v 4 - Check Admin->Jobs and see that many ES index jobs are queued 5 - Apply patch 6 - perl misc/link_bibs_to_authorities.pl -v 7 - Check Admin->Jobs and see that 1 index per commit from is enqueued -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32250 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |nick@bywatersolutions.com |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32250 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au --- Comment #3 from David Cook <dcook@prosentient.com.au> --- Looking good at a glance, and probably fixes some possible race conditions... -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32250 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david@davidnind.com --- Comment #4 from David Nind <david@davidnind.com> --- I've tested (using KTD with the sample data and ES7), and I think I have run through the test plan correctly. The changes in the first patch all seemed to work correctly - no more holds queue updates. For the second patch, for step 7, two jobs appear in the 'Current jobs only' list and don't disappear - even after leaving for about 10 minutes or so (I tried a couple of times, I also flushed the cache and ran restart_all). These are the entries in the current jobs table: 234 New null/1 Update Elasticsearch index 01/17/2023 01:16 233 New null/1 Update Elasticsearch index 01/17/2023 01:16 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32250 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32250 Emily Lamancusa <emily.lamancusa@montgomerycountymd.gov> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |emily.lamancusa@montgomeryc | |ountymd.gov -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32250 --- Comment #5 from Nick Clemens <nick@bywatersolutions.com> --- The last patch on bug 32594 will fix the ES job statuses -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32250 --- Comment #6 from David Nind <david@davidnind.com> --- I had another go at testing, but the jobs list doesn't seem to be uodating properly for me. Here is how I tested using KTD: 1. master is up-to-date, so the latest patches for bug 32594 are included. 2. Using ES7 with Searchengine set to Elasticsearch, rebuilt search index (koha-elasticsearch --rebuild -d -b -a kohadev). 3. Applied both patches, flush_memcached, restart_all. 4. Enabled RealTimeHoldsQueue system preference. 5. Ran perl misc/link_bibs_to_authorities.pl -v 6. Current job list look like this (no change after 10 minutes): 1 New null/1 Update Elasticsearch index 03/28/2023 17:18 2 New null/1 Update Elasticsearch index 03/28/2023 17:18 7. Changed LinkerModule system preference. 8. Ran perl misc/link_bibs_to_authorities.pl -v 9. There are now four jobs in the current job list: 1 New null/1 Update Elasticsearch index 03/28/2023 17:18 2 New null/1 Update Elasticsearch index 03/28/2023 17:18 4 New null/1 Update Elasticsearch index 03/28/2023 17:28 5 New null/1 Update Elasticsearch index 03/28/2023 17:28 Is there something else I need to do? It seems to me that either the ES index updates are not actually running, or that the status is not updating when it is finished (even though the latest patches for bug 32594 are in master). -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32250 --- Comment #7 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 149969 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=149969&action=edit Bug 32250: (follow-up) Commit background job to the DB We can't run a background job if it isn't in the database, however, this script runs with AutoCommit disabled. We need to enable it while generating the background job, then disable for the updates. I don't nkow if using a transaction would be preferable. I tried to solve this independently, but it requires consolidating the index requests to make this work easily -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32250 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|normal |major --- Comment #8 from Nick Clemens <nick@bywatersolutions.com> --- Raising severity as the linker currently doesn't successfully index for Elasticsearch due to a race condition -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32250 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32250 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #145305|0 |1 is obsolete| | --- Comment #9 from David Nind <david@davidnind.com> --- Created attachment 150100 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=150100&action=edit Bug 32250: Linking authorities does not change holdability If the only change is a linked heading, we don't need to rebuild the holds queue To test: 1 - Set preference RealTimeHoldsQueue to enable 2 - Run link_bibs_to_authorities 3 - Note holds queue jobs generated 4 - Apply patch 5 - repeat 6 - No holds queue updates Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32250 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #145319|0 |1 is obsolete| | --- Comment #10 from David Nind <david@davidnind.com> --- Created attachment 150101 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=150101&action=edit Bug 32250: Only reindex records when commiting This patch adds an array to catch updated bibs, and defers indexing until a batch of changes is committed To test: 1 - Set LinkerModule system preference to either first or last match Alternate this between runs of the linker to ensure changes are made 2 - Set SearchEngine to Elasticsearch and reindex (to ensure index is built) 3 - perl misc/link_bibs_to_authorities.pl -v 4 - Check Admin->Jobs and see that many ES index jobs are queued 5 - Apply patch 6 - perl misc/link_bibs_to_authorities.pl -v 7 - Check Admin->Jobs and see that 1 index per commit from is enqueued Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32250 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #149969|0 |1 is obsolete| | --- Comment #11 from David Nind <david@davidnind.com> --- Created attachment 150102 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=150102&action=edit Bug 32250: (follow-up) Commit background job to the DB We can't run a background job if it isn't in the database, however, this script runs with AutoCommit disabled. We need to enable it while generating the background job, then disable for the updates. I don't nkow if using a transaction would be preferable. I tried to solve this independently, but it requires consolidating the index requests to make this work easily Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32250 --- Comment #12 from David Nind <david@davidnind.com> --- How I tested (using KTD): 1. Start KTD with Elastic search (ktd --es7 up). 2. Update bibliographic record number 73 to remove one of the 010 LCCN numbers. Otherwise there is an error when running perl misc/link_bibs_to_authorities.pl -v: C4::Biblio::_koha_modify_biblioitem_nonmarc(): DBI Exception: DBD::mysql::st execute failed: Data too long for column 'lccn' at row 1 at /kohadevbox/koha/C4/Biblio.pm line 432 3. Set the RealTimeHoldsQueue system preference to Enable. 4. Run the authorities linker: perl misc/link_bibs_to_authorities.pl -v 5. Check the jobs list (Administration > Jobs > Manage jobs). Note that holds queue jobs were generated. 6. Set the LinkerModule system preference to either first or last match - alternate this between runs of the linker to ensure changes are made. 7. Rebuild search index (koha-elasticsearch --rebuild -d -b -a kohadev). 8. Run the authorities linker: perl misc/link_bibs_to_authorities.pl -v 9. Check the jobs list (Administration > Jobs > Manage jobs). Note that many Elasticsearch index jobs are queued. 10. Apply the patches, flush_memcached, restart_all. 11. Change the Set LinkerModule system preference. 12. Run the authorities linker: perl misc/link_bibs_to_authorities.pl -v 13. Check the jobs list (Administration > Jobs > Manage jobs). Note that: - There are no hold queue updates - That are a smaller number of Elastic search update (3 for me, instead of an update for each record) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32250 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart+koha@gmail. | |com --- Comment #13 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- I think we should remove AutoCommit here, see bug 27245. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32250 --- Comment #14 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 150239 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=150239&action=edit Bug 32250: (follow-up) Remove use of AutoCommit=0 This patch moves to using txn_begin and txn_commit -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32250 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Failed QA --- Comment #15 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- What about this one? 88 $dbh->commit(); -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32250 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32250 --- Comment #16 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 150293 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=150293&action=edit Bug 32250: (follow-up) Remove one more dbh commit and don't start a new transaction when done -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32250 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32250 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #150100|0 |1 is obsolete| | Attachment #150101|0 |1 is obsolete| | Attachment #150102|0 |1 is obsolete| | Attachment #150239|0 |1 is obsolete| | Attachment #150293|0 |1 is obsolete| | --- Comment #17 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 150501 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=150501&action=edit Bug 32250: Linking authorities does not change holdability If the only change is a linked heading, we don't need to rebuild the holds queue To test: 1 - Set preference RealTimeHoldsQueue to enable 2 - Run link_bibs_to_authorities 3 - Note holds queue jobs generated 4 - Apply patch 5 - repeat 6 - No holds queue updates Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32250 --- Comment #18 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 150502 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=150502&action=edit Bug 32250: Only reindex records when commiting This patch adds an array to catch updated bibs, and defers indexing until a batch of changes is committed To test: 1 - Set LinkerModule system preference to either first or last match Alternate this between runs of the linker to ensure changes are made 2 - Set SearchEngine to Elasticsearch and reindex (to ensure index is built) 3 - perl misc/link_bibs_to_authorities.pl -v 4 - Check Admin->Jobs and see that many ES index jobs are queued 5 - Apply patch 6 - perl misc/link_bibs_to_authorities.pl -v 7 - Check Admin->Jobs and see that 1 index per commit from is enqueued Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32250 --- Comment #19 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 150503 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=150503&action=edit Bug 32250: (follow-up) Commit background job to the DB We can't run a background job if it isn't in the database, however, this script runs with AutoCommit disabled. We need to enable it while generating the background job, then disable for the updates. I don't nkow if using a transaction would be preferable. I tried to solve this independently, but it requires consolidating the index requests to make this work easily Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32250 --- Comment #20 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 150504 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=150504&action=edit Bug 32250: (follow-up) Remove use of AutoCommit=0 This patch moves to using txn_begin and txn_commit Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> Bug 32250: (follow-up) Remove one more dbh commit and don't start a new transaction when done Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32250 --- Comment #21 from David Cook <dcook@prosentient.com.au> --- Just double checking... this is an Elasticsearch issue only right? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32250 --- Comment #22 from David Nind <david@davidnind.com> --- (In reply to David Cook from comment #21)
Just double checking... this is an Elasticsearch issue only right?
Yes - see step 2 of the 'Only reindex records when commiting' patch. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32250 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |23.05.00 released in| | Status|Passed QA |Pushed to master -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32250 --- Comment #23 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Pushed to master for 23.05. Nice work everyone, thanks! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32250 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to master |Pushed to stable Version(s)|23.05.00 |23.05.00,22.11.06 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32250 --- Comment #24 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Many hands makes light work, thankyou everyone! Pushed to 22.11.x for the next release -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32250 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lucas@bywatersolutions.com --- Comment #25 from Lucas Gass <lucas@bywatersolutions.com> --- This patchset does not apply cleanly to 22.05.x, no backport unless rebased. -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org