[Bug 42669] New: es_indexer_daemon.pl silently marks jobs finished on indexing failure and doesn't recover from NoNodes
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42669 Bug ID: 42669 Summary: es_indexer_daemon.pl silently marks jobs finished on indexing failure and doesn't recover from NoNodes Initiative type: --- Sponsorship --- status: Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: major Priority: P5 - low Component: Searching - Elasticsearch Assignee: koha-bugs@lists.koha-community.org Reporter: tomascohen@gmail.com QA Contact: testopia@bugs.koha-community.org Blocks: 39769 es_indexer_daemon.pl has two issues that cause silent data loss: 1. When update_index() throws an exception (e.g. NoNodes, BadResponse), the exception is caught and logged, but the job is still marked status='finished'. This means records are never re-indexed and silently disappear from search results. 2. The Search::Elasticsearch client (cxn_pool=Static) marks nodes as 'dead' after a connection failure and applies exponential backoff up to 60 minutes. Since the indexer objects are created once at daemon startup, a brief ES restart (e.g. Docker upgrade) leaves the daemon permanently stuck in NoNodes state until manually restarted. Fix: - On NoNodes: abort the batch, recreate the indexer object (resetting the connection pool), and reset jobs to status='new' for automatic retry. - On other indexing errors: mark jobs as 'failed' instead of 'finished'. Test plan: 1. Start es_indexer_daemon.pl 2. Stop Elasticsearch 3. Add/modify a biblio record (triggers indexing job) 4. Verify the job stays in status='new' (not 'finished') 5. Restart Elasticsearch 6. Verify the job is picked up and indexed successfully Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39769 [Bug 39769] es_indexer_daemon.pl uses stale L1 cache -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42669 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED CC| |gabrielmunoz@theke.io, | |nick@bywatersolutions.com, | |tomascohen@gmail.com Assignee|koha-bugs@lists.koha-commun |tomascohen@gmail.com |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42669 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Needs Signoff Patch complexity|--- |Trivial patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42669 --- Comment #1 from Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> --- Created attachment 199401 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199401&action=edit Bug 42669: Fix es_indexer_daemon.pl silent job failure and NoNodes recovery When update_index() throws, the daemon now: - On NoNodes: aborts the batch, recreates the indexer object to reset the dead-node connection pool, and resets jobs to 'new' for retry. - On other errors: marks jobs as 'failed' instead of 'finished'. This prevents silent data loss where records disappear from search results because their indexing jobs were incorrectly marked as done. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42669 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #199401|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42669 --- Comment #2 from Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> --- Created attachment 199402 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199402&action=edit Bug 42669: Fix es_indexer_daemon.pl silent job failure and NoNodes recovery When update_index() throws, the daemon now: - On NoNodes: aborts the batch, recreates the indexer object to reset the dead-node connection pool, and resets jobs to 'new' for retry. - On other errors: marks jobs as 'failed' instead of 'finished'. This prevents silent data loss where records disappear from search results because their indexing jobs were incorrectly marked as done. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42669 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the| |The Elasticsearch indexer release notes| |daemon | |(es_indexer_daemon.pl) was | |silently marking background | |indexing jobs as "finished" | |even when Elasticsearch was | |unreachable, causing | |records to disappear from | |search results with no | |indication of failure. | |Additionally, after a brief | |ES outage (e.g. a Docker | |restart), the daemon's | |connection pool would mark | |the node as dead with | |exponential backoff, | |leaving it permanently | |stuck until manually | |restarted. The daemon now | |resets jobs to "new" on | |NoNodes errors so they are | |automatically retried once | |connectivity is restored, | |recreates the ES client to | |reset the connection pool, | |and correctly marks jobs as | |"failed" for other indexing | |errors. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42669 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |kyle@bywatersolutions.com, | |mark@hofstetter.at -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42669 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42669 --- Comment #4 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 201879 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201879&action=edit Bug 42669: Use exception type check for NoNodes and don't mask real failures Two follow-up fixes from QA review of the previous commit: - Detect Search::Elasticsearch::Error::NoNodes via ref($_) rather than regex-matching the stringified exception, matching the existing pattern already used in about.pl. - Only reset jobs to 'new' for retry when NoNodes is the sole problem. If a genuine indexing error also occurred in the same batch, record the batch as 'failed' instead of silently discarding that error via the NoNodes retry path. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42669 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #199402|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42669 --- Comment #3 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 201878 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201878&action=edit Bug 42669: Fix es_indexer_daemon.pl silent job failure and NoNodes recovery When update_index() throws, the daemon now: - On NoNodes: aborts the batch, recreates the indexer object to reset the dead-node connection pool, and resets jobs to 'new' for retry. - On other errors: marks jobs as 'failed' instead of 'finished'. This prevents silent data loss where records disappear from search results because their indexing jobs were incorrectly marked as done. Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42669 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42669 --- Comment #7 from Lisette Scheer <lisette@bywatersolutions.com> --- Created attachment 201890 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201890&action=edit Bug 42669: Use exception type check for NoNodes and don't mask real failures Two follow-up fixes from QA review of the previous commit: - Detect Search::Elasticsearch::Error::NoNodes via ref($_) rather than regex-matching the stringified exception, matching the existing pattern already used in about.pl. - Only reset jobs to 'new' for retry when NoNodes is the sole problem. If a genuine indexing error also occurred in the same batch, record the batch as 'failed' instead of silently discarding that error via the NoNodes retry path. Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42669 --- Comment #6 from Lisette Scheer <lisette@bywatersolutions.com> --- Created attachment 201889 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201889&action=edit Bug 42669: Fix es_indexer_daemon.pl silent job failure and NoNodes recovery When update_index() throws, the daemon now: - On NoNodes: aborts the batch, recreates the indexer object to reset the dead-node connection pool, and resets jobs to 'new' for retry. - On other errors: marks jobs as 'failed' instead of 'finished'. This prevents silent data loss where records disappear from search results because their indexing jobs were incorrectly marked as done. Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42669 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA --- Comment #5 from Lisette Scheer <lisette@bywatersolutions.com> --- Works great! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42669 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #201878|0 |1 is obsolete| | Attachment #201879|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42669 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |lisette@bywatersolutions.co |y.org |m CC| |lisette@bywatersolutions.co | |m -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42669 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to main Version(s)| |26.11.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42669 --- Comment #8 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Thanks everyone! Pushed to main for 26.11! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42669 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |rel_25_11_candidate, | |rel_26_05_candidate -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42669 Lucas Gass (lukeg) <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|26.11.00 |26.11.00,26.05.03 released in| | Status|Pushed to main |Pushed to stable -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42669 --- Comment #9 from Lucas Gass (lukeg) <lucas@bywatersolutions.com> --- Nice work everyone! Pushed to 26.05.x for the upcoming 26.05.03 release. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42669 Lucas Gass (lukeg) <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lucas@bywatersolutions.com Keywords|rel_26_05_candidate | -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org