[Bug 22828] New: Elasticsearch - display errors encountered during indexing on the command line
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22828 Bug ID: 22828 Summary: Elasticsearch - display errors encountered during indexing on the command line Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Searching - Elasticsearch Assignee: koha-bugs@lists.koha-community.org Reporter: nick@bywatersolutions.com Target Milestone: --- Currently any records that do not index simply fail silently To recreate: 1 - Use the Koha sample data, or insert a blank 245$b into a record (easiest way is using advanced cataloging editor 2 - Reindex elasticsearch 3 - Check the ES count on the about page 4 - Check the count in the DB (SELECT count(*) FROM biblio) 5 - They don't match! 6 - perl misc/search_tools/rebuild_elastic_search.pl -v -v 7 - No errors indicated -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22828 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|--- |Trivial patch Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22828 --- Comment #1 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 89226 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=89226&action=edit Bug 22828: Elasticsearch - display errors encountered during indexing on the command line To test: 1 - Use the Koha sample data, or insert a blank 245$b into a record (easiest way is using advanced cataloging editor 2 - Reindex elasticsearch 3 - Check the ES count on the about page 4 - Check the count in the DB (SELECT count(*) FROM biblio) 5 - They don't match! 6 - perl misc/search_tools/rebuild_elastic_search.pl -v -v 7 - No errors indicated 8 - Apply patch 9 - perl misc/search_tools/rebuild_elastic_search.pl -v 10 - You should be notified of an error 11 - perl misc/search_tools/rebuild_elastic_search.pl -v -v 12 - You should be notified of the specific biblio with an error and a (somewhat) readable reason 13 - perl misc/search_tools/rebuild_elastic_search.pl 14 - No output -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22828 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #89226|0 |1 is obsolete| | --- Comment #2 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 89228 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=89228&action=edit Bug 22828: Elasticsearch - display errors encountered during indexing on the command line To test: 1 - Use the Koha sample data, or insert a blank 245$b into a record (easiest way is using advanced cataloging editor 2 - Reindex elasticsearch 3 - Check the ES count on the about page 4 - Check the count in the DB (SELECT count(*) FROM biblio) 5 - They don't match! 6 - perl misc/search_tools/rebuild_elastic_search.pl -v -v 7 - No errors indicated 8 - Apply patch 9 - perl misc/search_tools/rebuild_elastic_search.pl -v 10 - You should be notified of an error 11 - perl misc/search_tools/rebuild_elastic_search.pl -v -v 12 - You should be notified of the specific biblio with an error and a (somewhat) readable reason 13 - perl misc/search_tools/rebuild_elastic_search.pl 14 - No output -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22828 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ere.maijala@helsinki.fi, | |glasklas@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22828 --- Comment #3 from Ere Maijala <ere.maijala@helsinki.fi> --- I agree errors should be displayed, but I don't think we should fail indexing a record with an empty field. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22828 Ere Maijala <ere.maijala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=22829 -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22828 Michal Denar <black23@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |black23@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22828 --- Comment #4 from Nick Clemens <nick@bywatersolutions.com> --- (In reply to Ere Maijala from comment #3)
I agree errors should be displayed, but I don't think we should fail indexing a record with an empty field.
I don't know what I meant there, but I cannot recreate a record that will not index, at least not by simply adding an empty field. All sample records index successfully now. I don't think this patch is a bad idea still in the case that we do encounter errors -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22828 --- Comment #5 from Ere Maijala <ere.maijala@helsinki.fi> --- Agreed. Could you rebase, please? -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22828 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #89228|0 |1 is obsolete| | --- Comment #6 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 95096 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=95096&action=edit Bug 22828: Elasticsearch - display errors encountered during indexing on the command line To test: 1 - Use the Koha sample data, or insert a blank 245$b into a record (easiest way is using advanced cataloging editor 2 - Reindex elasticsearch 3 - Check the ES count on the about page 4 - Check the count in the DB (SELECT count(*) FROM biblio) 5 - They don't match! 6 - perl misc/search_tools/rebuild_elasticsearch.pl -v -v 7 - No errors indicated 8 - Apply patch 9 - perl misc/search_tools/rebuild_elasticsearch.pl -v 10 - You should be notified of an error 11 - perl misc/search_tools/rebuild_elasticsearch.pl -v -v 12 - You should be notified of the specific biblio with an error and a (somewhat) readable reason 13 - perl misc/search_tools/rebuild_elasticsearch.pl 14 - No output -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22828 Ere Maijala <ere.maijala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #95096|0 |1 is obsolete| | --- Comment #7 from Ere Maijala <ere.maijala@helsinki.fi> --- Created attachment 95098 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=95098&action=edit Bug 22828: Elasticsearch - display errors encountered during indexing on the command line To test: 1 - Use the Koha sample data, or insert a blank 245$b into a record (easiest way is using advanced cataloging editor 2 - Reindex elasticsearch 3 - Check the ES count on the about page 4 - Check the count in the DB (SELECT count(*) FROM biblio) 5 - They don't match! 6 - perl misc/search_tools/rebuild_elastic_search.pl -v -v 7 - No errors indicated 8 - Apply patch 9 - perl misc/search_tools/rebuild_elastic_search.pl -v 10 - You should be notified of an error 11 - perl misc/search_tools/rebuild_elastic_search.pl -v -v 12 - You should be notified of the specific biblio with an error and a (somewhat) readable reason 13 - perl misc/search_tools/rebuild_elastic_search.pl 14 - No output Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi> -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22828 Marjorie Barry-Vila <marjorie.barry-vila@collecto.ca> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |marjorie.barry-vila@collect | |o.ca -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22828 Séverine Queune <severine.queune@bulac.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22828 Séverine Queune <severine.queune@bulac.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #95098|0 |1 is obsolete| | --- Comment #8 from Séverine Queune <severine.queune@bulac.fr> --- Created attachment 95530 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=95530&action=edit Bug 22828: Elasticsearch - display errors encountered during indexing on the command line To test: 1 - Use the Koha sample data, or insert a blank 245$b into a record (easiest way is using advanced cataloging editor 2 - Reindex elasticsearch 3 - Check the ES count on the about page 4 - Check the count in the DB (SELECT count(*) FROM biblio) 5 - They don't match! 6 - perl misc/search_tools/rebuild_elastic_search.pl -v -v 7 - No errors indicated 8 - Apply patch 9 - perl misc/search_tools/rebuild_elastic_search.pl -v 10 - You should be notified of an error 11 - perl misc/search_tools/rebuild_elastic_search.pl -v -v 12 - You should be notified of the specific biblio with an error and a (somewhat) readable reason 13 - perl misc/search_tools/rebuild_elastic_search.pl 14 - No output Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi> Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr> -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22828 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@bugs.koha-c | |ommunity.org Assignee|koha-bugs@lists.koha-commun |nick@bywatersolutions.com |ity.org | --- Comment #9 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Do we really want to continue without adding tests? -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22828 --- Comment #10 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to Nick Clemens from comment #0)
Currently any records that do not index simply fail silently
To recreate: 1 - Use the Koha sample data, or insert a blank 245$b into a record (easiest way is using advanced cataloging editor 2 - Reindex elasticsearch 3 - Check the ES count on the about page 4 - Check the count in the DB (SELECT count(*) FROM biblio) 5 - They don't match!
They match for me. Which record is supposed to be wrong in the sample data? I added a blank 245$b but the reindex is ok. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22828 Bouzid Fergani <bouzid.fergani@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #95530|0 |1 is obsolete| | --- Comment #11 from Bouzid Fergani <bouzid.fergani@inlibro.com> --- Created attachment 99040 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=99040&action=edit Bug 22828: Elasticsearch - display errors encountered during indexing on the command line To test: 1 - Use the Koha sample data, or insert a blank 245$b into a record (easiest way is using advanced cataloging editor 2 - Reindex elasticsearch 3 - Check the ES count on the about page 4 - Check the count in the DB (SELECT count(*) FROM biblio) 5 - They don't match! 6 - perl misc/search_tools/rebuild_elastic_search.pl -v -v 7 - No errors indicated 8 - Apply patch 9 - perl misc/search_tools/rebuild_elastic_search.pl -v 10 - You should be notified of an error 11 - perl misc/search_tools/rebuild_elastic_search.pl -v -v 12 - You should be notified of the specific biblio with an error and a (somewhat) readable reason 13 - perl misc/search_tools/rebuild_elastic_search.pl 14 - No output Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi> Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr> Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22828 Bouzid Fergani <bouzid.fergani@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bouzid.fergani@inlibro.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22828 --- Comment #12 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 99388 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=99388&action=edit Bug 22828: Add tests -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22828 --- Comment #13 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- kohadev-koha@df2c0b4bf86c:/kohadevbox/koha$ perl misc/search_tools/rebuild_elasticsearch.pl -v [9614] Checking state of biblios index [9614] Checking state of authorities index [9614] Indexing biblios [9614] Committing final records... [9614] There were errors during indexing [9614] Total 436 records indexed [9614] Indexing authorities [9614] 1000 records processed [9614] Committing final records... [9614] Total 1706 records indexed with -v -v the output is not readable and the error is lost in the middle of the id. Should not we have it at the end? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22828 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Failed QA --- Comment #14 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Waiting for an answer. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22828 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Signed Off --- Comment #15 from Nick Clemens <nick@bywatersolutions.com> --- (In reply to Jonathan Druart from comment #14)
Waiting for an answer.
I cannot recreate an error at this time, so I am not sure what you mean. (In reply to Jonathan Druart from comment #13)
with -v -v the output is not readable and the error is lost in the middle of the id. Should not we have it at the end?
Can you show a sample and or what you propose? I don't think it actually matters where it falls. I expect in practice the output will be sent to a file. I am not saying there isn't room for improvement, but ensuring they are returned in any form is better than ignoring them as we do now -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22828 --- Comment #16 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Let improve later then! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22828 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22828 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #99040|0 |1 is obsolete| | Attachment #99388|0 |1 is obsolete| | --- Comment #17 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 103645 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=103645&action=edit Bug 22828: Elasticsearch - display errors encountered during indexing on the command line To test: 1 - Use the Koha sample data, or insert a blank 245$b into a record (easiest way is using advanced cataloging editor 2 - Reindex elasticsearch 3 - Check the ES count on the about page 4 - Check the count in the DB (SELECT count(*) FROM biblio) 5 - They don't match! 6 - perl misc/search_tools/rebuild_elastic_search.pl -v -v 7 - No errors indicated 8 - Apply patch 9 - perl misc/search_tools/rebuild_elastic_search.pl -v 10 - You should be notified of an error 11 - perl misc/search_tools/rebuild_elastic_search.pl -v -v 12 - You should be notified of the specific biblio with an error and a (somewhat) readable reason 13 - perl misc/search_tools/rebuild_elastic_search.pl 14 - No output Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi> Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr> Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com> Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22828 --- Comment #18 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 103646 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=103646&action=edit Bug 22828: Add tests Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22828 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to master Version(s)| |20.05.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22828 --- Comment #19 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Nice work everyone! Pushed to master for 20.05 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22828 Joy Nelson <joy@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|20.05.00 |20.05.00, 19.11.06 released in| | CC| |joy@bywatersolutions.com Status|Pushed to master |Pushed to stable --- Comment #20 from Joy Nelson <joy@bywatersolutions.com> --- Backported to 19.11.x for 19.11.06 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22828 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to stable |Pushed to oldstable CC| |lucas@bywatersolutions.com Version(s)|20.05.00, 19.11.06 |20.05.00, 19.11.06, released in| |19.05.11 --- Comment #21 from Lucas Gass <lucas@bywatersolutions.com> --- backported to 19.05.x for 19.05.11 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22828 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |martin.renvoize@ptfs-europe | |.com Summary|Elasticsearch - display |Add display of errors |errors encountered during |encountered during indexing |indexing on the command |on the command line |line | -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org