[Bug 39076] New: Elasticsearch timeouts when committing import batches
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39076 Bug ID: 39076 Summary: Elasticsearch timeouts when committing import batches Change sponsored?: --- Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Tools Assignee: koha-bugs@lists.koha-community.org Reporter: nick@bywatersolutions.com QA Contact: testopia@bugs.koha-community.org We have seen in production that the ES server can get overwhelmed and start issuing timeouts during imports. The timeout errors often come from authorities linking during the bib import process -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39076 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39076 --- Comment #1 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 177692 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=177692&action=edit Bug 39076: Use state variable to avoid creating extra ES connections As elsewhere in the code, we should use a state variable for our 'bib_searcher' object to avoid overhead of extra ES connections. Worth noting - we don't actually use the bib_search at all during record auto linking as we pass skipmetadata To test: 0 - Apply patch 1 - Enable biblio auto linking AutoLinkBiblios, CatalogModuleRelink, LinkerRelink 2 - Stage and import a marc file 3 - Confirm record authorities link as expected 4 - Search authorities in the staff interface 5 - Confirm usage counts are shown for results -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39076 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |nick@bywatersolutions.com |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39076 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39076 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #177692|0 |1 is obsolete| | --- Comment #2 from Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> --- Created attachment 177695 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=177695&action=edit Bug 39076: Use state variable to avoid creating extra ES connections As elsewhere in the code, we should use a state variable for our 'bib_searcher' object to avoid overhead of extra ES connections. Worth noting - we don't actually use the 'bib_searcher' at all during record auto linking as we pass skipmetadata To test: 0 - Apply patch 1 - Enable biblio auto linking AutoLinkBiblios, CatalogModuleRelink, LinkerRelink 2 - Stage and import a marc file 3 - Confirm record authorities link as expected 4 - Search authorities in the staff interface 5 - Confirm usage counts are shown for results Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39076 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |tomascohen@gmail.com --- Comment #3 from Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> --- It doesn't break current behavior. I tested reindexing, searching and run the SearchEngine related tests. QA scripts happy. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39076 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au --- Comment #4 from David Cook <dcook@prosentient.com.au> --- So I was tempted to say this isn't really a Koha-ish way of doing things... but it turns out that it is! It's already done heaps in C4/Heading.pm and C4/Matcher.pm Normally, I think that we'd just store the Koha::SearchEngine::Elasticsearch::Search object in $self (as an attribute of Koha::SearchEngine::Search that is), but that would take more code. It might be easier to understand for someone unfamiliar with "state", but it's less elegant for sure. While "state" seems like it could be abused in some scenarios, I think that this one is a pretty reasonable place to use it. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39076 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39076 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #177695|0 |1 is obsolete| | --- Comment #5 from David Cook <dcook@prosentient.com.au> --- Created attachment 177714 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=177714&action=edit Bug 39076: Use state variable to avoid creating extra ES connections As elsewhere in the code, we should use a state variable for our 'bib_searcher' object to avoid overhead of extra ES connections. Worth noting - we don't actually use the 'bib_searcher' at all during record auto linking as we pass skipmetadata To test: 0 - Apply patch 1 - Enable biblio auto linking AutoLinkBiblios, CatalogModuleRelink, LinkerRelink 2 - Stage and import a marc file 3 - Confirm record authorities link as expected 4 - Search authorities in the staff interface 5 - Confirm usage counts are shown for results Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: David Cook <dcook@prosentient.com.au> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39076 --- Comment #6 from David Cook <dcook@prosentient.com.au> --- It passes all my manual and automated tests. If it does cause problems, it'll be easy enough to revert. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39076 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |dcook@prosentient.com.au |y.org | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39076 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |25.05.00 released in| | Status|Passed QA |Pushed to main -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39076 --- Comment #7 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Pushed for 25.05! Well done everyone, thank you! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39076 Miranda Nero <mnero@oslri.net> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mnero@oslri.net -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39076 --- Comment #8 from David Cook <dcook@prosentient.com.au> --- On a side note, does this fix your prod issue, Nick? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39076 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m Status|Pushed to main |Pushed to stable Version(s)|25.05.00 |25.05.00,24.11.10 released in| | --- Comment #9 from Fridolin Somers <fridolin.somers@biblibre.com> --- Pushed to 24.11.x for 24.11.10 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39076 Wainui Witika-Park <wainuiwitikapark@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to stable |Needs documenting --- Comment #10 from Wainui Witika-Park <wainuiwitikapark@catalyst.net.nz> --- Not backporting to 22.11.x as it wasn't backported to 24.05.x -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon@bugs.koha-community.org