[Bug 44022] New: Elasticsearch: product check workaround is never restored, one failed request breaks the client for the rest of the process
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=44022 Bug ID: 44022 Summary: Elasticsearch: product check workaround is never restored, one failed request breaks the client for the rest of the process Initiative type: --- Sponsorship --- status: Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Searching - Elasticsearch Assignee: koha-bugs@lists.koha-community.org Reporter: johan@imcode.com QA Contact: testopia@bugs.koha-community.org Target Milestone: --- Bug 38345 added a monkey patch of Search::Elasticsearch::Role::Cxn::process_response in Koha::SearchEngine::Elasticsearch. When a response has no x-elastic-product header, it assigns $Search::Elasticsearch::Role::Cxn::PRODUCT_CHECK_VALUE = ''; to skip the product check (needed for OpenSearch). The assignment is permanent. A client side failure (timeout, refused or reset connection) is reported by HTTP::Tiny as a synthetic 599 response without headers, so it triggers the assignment too. From then on the product check expects an empty product header, and every response from a healthy Elasticsearch node, which does send x-elastic-product: Elasticsearch, is rejected with ProductCheck. The node is marked dead and every following request fails with "No nodes are available" (NoNodes), although Elasticsearch is reachable again. A fresh process works, so this only affects long running processes: es_indexer_daemon.pl and Plack workers. They stay broken until restarted. With several nodes configured, failover fails the same way: a failed first node poisons the process and the healthy second node is then rejected. To reproduce: 1. Start a long running process that talks to Elasticsearch, for example misc/workers/es_indexer_daemon.pl, or a script that creates a Koha::SearchEngine::Indexer in a loop and calls update_index(). 2. Stop Elasticsearch for about ten seconds, then start it again. Result: the process keeps failing with NoNodes after Elasticsearch is back, while a new process can reach it. Expected: the process recovers as soon as Elasticsearch answers. Fix: use local, so that the product check is skipped for that one response only. This keeps the OpenSearch behaviour of bug 38345. The patch adds t/Koha/SearchEngine/Elasticsearch/ProductCheck.t, which needs no running Elasticsearch. It fails on the current code and passes with the patch. This is also a prerequisite for recovering from an outage in a long running process (see the follow-up bugs about es_indexer_daemon.pl). -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=44022 Johan Larsson <johan@imcode.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=38345 -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=44022 --- Comment #1 from Johan Larsson <johan@imcode.com> --- Created attachment 210825 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210825&action=edit Bug 44022: Do not disable the Elasticsearch product check for the rest of the process Bug 38345 made Koha work with OpenSearch by wrapping Search::Elasticsearch::Role::Cxn::process_response and setting $Search::Elasticsearch::Role::Cxn::PRODUCT_CHECK_VALUE = '' whenever a response has no x-elastic-product header, to skip the product check. The assignment is permanent. A client side failure (timeout, refused or reset connection) is reported by HTTP::Tiny as a synthetic 599 response without headers, so after the first such failure the setting is changed for the whole process. In a long running process (es_indexer_daemon.pl, Plack workers) the next response from a healthy Elasticsearch node is then rejected, the node is marked dead, and every following request fails with 'No nodes are available' until the process is restarted, even though Elasticsearch is reachable again. With several nodes configured failover fails the same way. Use local, so the product check is skipped for that one response only. Test plan: 1. Start a long running process that talks to Elasticsearch, e.g. misc/workers/es_indexer_daemon.pl, or use a script that creates a Koha::SearchEngine::Indexer in a loop and calls update_index(). 2. Stop Elasticsearch for about ten seconds, then start it again. => Without the patch the process keeps failing with 'No nodes are available' (type NoNodes) after Elasticsearch is back; a new process can reach Elasticsearch. => With the patch the process recovers as soon as Elasticsearch answers. 3. prove t/Koha/SearchEngine/Elasticsearch/ProductCheck.t -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=44022 Johan Larsson <johan@imcode.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |44024 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=44024 [Bug 44024] es_indexer_daemon.pl: keep the batch and retry while Elasticsearch is unreachable, jobs are lost with STOMP -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=44022 Jonathan Druart <jonathan.druart@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@gmail.com --- Comment #2 from Jonathan Druart <jonathan.druart@gmail.com> --- If this is ready for testing please set the status to "Needs Signoff". -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=44022 Jonathan Druart <jonathan.druart@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |johan@imcode.com |ity.org | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=44022 Johan Larsson <johan@imcode.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=44022 Johan Larsson <johan@imcode.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |44025 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=44025 [Bug 44025] Elasticsearch: use all addresses of a single <server> host name as nodes, so the client can fail over -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=44022 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david@davidnind.com --- Comment #3 from David Nind <david@davidnind.com> --- Some comments on the test plan: 1. Step 1: For "normal" (non-developer) people to test this patch, then some more instructions on how to do this, or a script and details on how to set it up, are required. 2. Step 2: The command (or package command) on how to do this is required. Apologies for being a bit pedantic - I'm seeing this more and more: developers and system administrators are probably really familiar with how to do these things, us normal (non-developer) people, not so much! David -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon@bugs.koha-community.org