[Bug 24438] New: index can fail on timeout
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24438 Bug ID: 24438 Summary: index can fail on timeout Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Searching - Elasticsearch Assignee: koha-bugs@lists.koha-community.org Reporter: nick@bywatersolutions.com We have seen occasional timeouts when indexing elasticsearch, these cause the indexing to stop and fail. We should handle the response as the code indicates: 102 sub update_index { 103 my ($self, $biblionums, $records) = @_; 104 105 my $conf = $self->get_elasticsearch_params(); 106 my $elasticsearch = $self->get_elasticsearch(); 107 my $documents = $self->marc_records_to_documents($records); 108 my @body; 109 110 for (my $i=0; $i < scalar @$biblionums; $i++) { 111 my $id = $biblionums->[$i]; 112 my $document = $documents->[$i]; 113 push @body, { 114 index => { 115 _id => $id 116 } 117 }; 118 push @body, $document; 119 } 120 if (@body) { 121 my $response = $elasticsearch->bulk( 122 index => $conf->{index_name}, 123 type => 'data', # is just hard coded in Indexer.pm? 124 body => \@body 125 ); 126 } 127 # TODO: handle response 128 return 1; 129 } -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24438 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |alex.arnaud@biblibre.com, | |ere.maijala@helsinki.fi -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24438 Björn Nylén <bjorn.nylen@ub.lu.se> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bjorn.nylen@ub.lu.se --- Comment #1 from Björn Nylén <bjorn.nylen@ub.lu.se> --- WE've been seeing the same. However we've recently tried to add a request_timeout parameter the the ES indexer in rebuild_elasticsearch.pl to work around the problem. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24438 --- Comment #2 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- 3 years later - does this problem still exist in our current implementation? -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24438 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=36702 CC| |kyle@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24438 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |kyle@bywatersolutions.com |ity.org | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24438 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |36702 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 [Bug 36702] Background Jobs: Add option to retry failed jobs with a max tries parameter -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24438 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on|36702 | Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 [Bug 36702] Background Jobs: Add option to retry failed jobs with a max tries parameter -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24438 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |36702 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36702 [Bug 36702] Background Jobs: Add option to retry failed jobs with a max tries parameter -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24438 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24438 --- Comment #3 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 201093 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201093&action=edit Bug 24438: Add a bulk_index_records method to Elasticsearch::Indexer This patch adds bulk_index_records: it indexes records in chunks, halves a batch Elasticsearch rejects as too large until it fits, and returns the ids that could not be indexed. The rebuild script and the indexer daemon both use it instead of each keeping their own copy. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24438 --- Comment #4 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 201094 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201094&action=edit Bug 24438: Add unit tests Test Plan: 1) Apply this patch 2) prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Indexer.t -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24438 --- Comment #5 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 201095 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201095&action=edit Bug 24438: Use bulk_index_records in the Elasticsearch indexer daemon This patch replaces the daemon's own batch-splitting loop with bulk_index_records. Records the bulk response reports as individual errors are now retried instead of only logged. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24438 --- Comment #6 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 201096 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201096&action=edit Bug 24438: Reindex script ( rebuild_elasticsearch.pl ) reports success even when commits fail When a commit fails ( for example a timeout ), rebuild_elasticsearch.pl logs the error, drops the batch, and still exits 0, so a reindex that lost records looks successful. This patch indexes each batch through bulk_index_records and exits non-zero if any records could not be indexed, including failures in forked children. When Elasticsearch is briefly overwhelmed it retries the failed records, waiting longer each attempt ( --retries, --retry-delay ), before giving up. Test Plan: 1) Apply the patches 2) prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Indexer.t 3) Run a reindex, note the output is unchanged and it exits 0: misc/search_tools/rebuild_elasticsearch.pl -v ; echo $? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24438 --- Comment #7 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 201097 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201097&action=edit Bug 24438: Make koha-elasticsearch report rebuild failures The koha-elasticsearch wrapper warns on a failed rebuild but always exits 0. This patch makes it exit non-zero so the failure is visible to cron or whatever called it. Test Plan: 1) Apply this patch 2) Force a rebuild failure for an instance ( e.g. point the elasticsearch server in koha-conf.xml at a dead host ) 3) koha-elasticsearch --rebuild <instance> ; echo $? 4) Note it now exits non-zero! -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org