[Bug 18624] New: Run time errors when searching authorities with Elastic search 5.3
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18624 Bug ID: 18624 Summary: Run time errors when searching authorities with Elastic search 5.3 Change sponsored?: --- Product: Koha Version: master Hardware: PC OS: Linux Status: NEW Severity: critical Priority: P5 - low Component: Searching Assignee: gmcharlt@gmail.com Reporter: david.holoshka@ub.lu.se QA Contact: testopia@bugs.koha-community.org Searching for a known authority entry (eg. Bear or Meyer) results in either the following runtime errors or no result (sort none). Sort on finds nothing Header search only $a: Software error: Invalid marclist field provided: all at /usr/local/koha/Koha/SearchEngine/Elasticsearch/QueryBuilder.pm line 433. Koha::SearchEngine::Elasticsearch::QueryBuilder::build_authorities_query_compat(Koha::SearchEngine::Elasticsearch::QueryBuilder=HASH(0x90e2110), ARRAY(0x90d4630), ARRAY(0x90d4660), ARRAY(0x914bf30), ARRAY(0x912f6b0), ARRAY(0x912f0f8), "", "") called at /usr/local/koha/authorities/authorities-home.pl line 93 Search Main Heading : Software error: Can't use string ("9932563") as an ARRAY ref while "strict refs" in use at /usr/local/koha/Koha/SearchEngine/Elasticsearch/Search.pm line 202. auth index entry in elastic search: { "_index" : "koha_ulibrary_authorities", "_type" : "data", "_id" : "1756", "_score" : 1.0, "_source" : { "Personal-name-see-also-from" : null, "Meeting-name-heading" : null, "Personal-name" : [ "Bear, Roberta Meyer" ], "Corporate-name-see-also-from" : null, "Heading-use-series-added-entry" : " ", "es_id" : "1756", "See-also-from" : null, "Meeting-name-see-also-from" : null, "LC-card-number" : null, "record" : [ [ "LDR", null, null, "_", "00262nz a2200121o 4500" ], [ "001", null, null, "_", "11072" ], [ "003", null, null, "_", "LDU" ], [ "005", null, null, "_", "20170405133903.0" ], [ "008", null, null, "_", "170405930116 a aabn a aaa" ], [ "040", " ", " ", "a", "L", "c", "L" ], [ "100", "1", " ", "a", "Bear, Roberta Meyer" ], [ "400", "1", " ", "a", "Meyer Bear, Roberta" ], [ "942", " ", " ", "a", "PERSO_NAME" ], [ "999", " ", " ", "c", "1756", "d", "1756" ] ], "Match-heading-see-from" : null, "record-source" : [ "L", "L" ], "Heading-use-main-or-added-entry" : " ", "Meeting-name-see-from" : null, "Personal-name-heading" : [ "Bear, Roberta Meyer" ], "Heading-use-subject-added-entry" : "a", "Personal-name-see-from" : [ "Meyer Bear, Roberta" ], "Local-number" : [ "11072" ], "Corporate-name-see-from" : null, "Match" : null, "authtype" : [ "PERSO_NAME" ], "Kind-of-record" : "1", "Match-heading" : [ "Bear, Roberta Meyer" ], "Descriptive-cataloging-rules" : "1", "See-from" : null, "Heading" : [ "Bear, Roberta Meyer" ], "Heading-Main" : null, "Subject-heading-thesaurus" : "6", "Meeting-name" : null } }, We are using the default field mapping. Koha running on Debian, I have applied a patch not yet on main (Bug 18434 - Elasticsearch indexing broken with newer catmandu version ) to make the biblio search work. Mysql ES 5.3 (it seems to be working fine) on oracle java 8 Marc 21 All perl catmandu modules are the latest from cpanm. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18624 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18624 --- Comment #1 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 64385 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=64385&action=edit Bug 18624 - Run time errors when searching authorities with Elastic search 5.3 It seems we have a syntax mismatch - any vs all all seems to the be list we expect so updating code that way To test: 1 - Enabled Elasticsearch 2 - Index some authorities 3 - Perform a 'Search entire record' search 4 - Internal server error ( Invalid marclist field provided: all at /usr/local/koha/Koha/SearchEngine/Elasticsearch/QueryBuilder.pm line 433. ) 5 - Apply patch 6 - Search should work -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18624 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nick@bywatersolutions.com --- Comment #2 from Nick Clemens <nick@bywatersolutions.com> --- This will take care of the first error at least. I can't recreate the "Can't use string ("9932563") as an ARRAY ref" - can you file that as a new bug with specific instructions for that issue? There are additional things you mention here, they should probably each go on a separate bug as well (no result, sort error, etc) it is easier to fix things one at a time :-) I am also adding a followup to 18434 as wee need to append entries for facets and sorting as well - i will be looking at fixing the tests / diagnosing why jonathan has differing results asap. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18624 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Run time errors when |Software error when |searching authorities with |searching authorities in |Elastic search 5.3 |Elasticsearch - incorrect | |parameter "any" should be | |"all" -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18624 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|gmcharlt@gmail.com |nick@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18624 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@bugs.koha-c | |ommunity.org --- Comment #3 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Nice, please provide tests. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18624 --- Comment #4 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Nick, please provide tests. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18624 --- Comment #5 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 65591 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=65591&action=edit Bug 18624: Minor refactoring to ease testing Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18624 --- Comment #6 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 65592 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=65592&action=edit Bug 18624: Regression test for 'any' vs 'all' This test makes it explicit that the only string producing _all as index on build_authorities_query_compat is 'all'. To test: - Apply this patch - Run: $ sudo koha-shell kohadev k$ cd kohaclone k$ prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t => FAIL: Test fails because the list of valid values is wrong in Koha. Note: this list has to be in sync with the templates passing the same values. A followup will be added to fix a discrepancy found between OPAC and Intranet. Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18624 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #64385|0 |1 is obsolete| | --- Comment #7 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 65593 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=65593&action=edit Bug 18624: Run time errors when searching authorities with Elastic search 5.3 It seems we have a syntax mismatch - any vs all all seems to the be list we expect so updating code that way To test: 1 - Enabled Elasticsearch 2 - Index some authorities 3 - Perform a 'Search entire record' search 4 - Internal server error ( Invalid marclist field provided: all at /usr/local/koha/Koha/SearchEngine/Elasticsearch/QueryBuilder.pm line 433. ) 5 - Run: $ sudo koha-shell kohadev k$ cd kohaclone k$ prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t => FAIL: Tests fail because 'any is used' 6 - Apply patch 7 - Search should work 8 - Run: k$ prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t => SUCCESS: Tests pass! 9 Sign off :-D Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Edited the test plan so it mentions the new tests -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18624 --- Comment #8 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 65594 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=65594&action=edit Bug 18624: (followup) opac-authorities-home.tt uses 'any' Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18624 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |tomascohen@gmail.com Patch complexity|--- |Small patch Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18624 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Hardware|PC |All CC| |m.de.rooy@rijksmuseum.nl OS|Linux |All -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18624 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18624 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #65591|0 |1 is obsolete| | --- Comment #9 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 65882 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=65882&action=edit Bug 18624: Minor refactoring to ease testing Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18624 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #65592|0 |1 is obsolete| | --- Comment #10 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 65883 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=65883&action=edit Bug 18624: Regression test for 'any' vs 'all' This test makes it explicit that the only string producing _all as index on build_authorities_query_compat is 'all'. To test: - Apply this patch - Run: $ sudo koha-shell kohadev k$ cd kohaclone k$ prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t => FAIL: Test fails because the list of valid values is wrong in Koha. Note: this list has to be in sync with the templates passing the same values. A followup will be added to fix a discrepancy found between OPAC and Intranet. Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18624 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #65593|0 |1 is obsolete| | --- Comment #11 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 65884 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=65884&action=edit Bug 18624: Run time errors when searching authorities with Elastic search 5.3 It seems we have a syntax mismatch - any vs all all seems to the be list we expect so updating code that way To test: 1 - Enabled Elasticsearch 2 - Index some authorities 3 - Perform a 'Search entire record' search 4 - Internal server error ( Invalid marclist field provided: all at /usr/local/koha/Koha/SearchEngine/Elasticsearch/QueryBuilder.pm line 433. ) 5 - Run: $ sudo koha-shell kohadev k$ cd kohaclone k$ prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t => FAIL: Tests fail because 'any is used' 6 - Apply patch 7 - Search should work 8 - Run: k$ prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t => SUCCESS: Tests pass! 9 Sign off :-D Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Edited the test plan so it mentions the new tests Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18624 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #65594|0 |1 is obsolete| | --- Comment #12 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 65885 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=65885&action=edit Bug 18624: (followup) opac-authorities-home.tt uses 'any' Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18624 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |m.de.rooy@rijksmuseum.nl |y.org | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18624 --- Comment #13 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- QA Comment: Trusting on Tomas for the ElasticSearch test. Code looks good to me. A unit test is added; it passes after the third patch. The Zebra side is fine too, also after the template change in the fourth patch, since AuthoritiesMarc does not care about all or any: else { # Assume any if no index was specified $attr = " \@attr 1=Any "; Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18624 --- Comment #14 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Side note: We are still waiting for the removal of the useless call of SearchAuthorities in QueryBuilder on bug 16976 in PNA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18624 --- Comment #15 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 66020 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=66020&action=edit Bug 18624: Do not return 1 in tests -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18624 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to Master --- Comment #16 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Pushed to master for 17.11, thanks to everybody involved! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18624 Fridolin SOMERS <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m Status|Pushed to Master |Pushed to Stable --- Comment #17 from Fridolin SOMERS <fridolin.somers@biblibre.com> --- Pushed to 17.05.x, will be in 17.05.04 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18624 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |katrin.fischer@bsz-bw.de --- Comment #18 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- These patches have been pushed to 16.11.x and will be in 16.11.12. Please note: Not all Elasticsearch related patches have been pushed to 16.11.x as there were some problems/conflicts and I received no answer on the bugs. This applied cleanly, so I have pushed it. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18624 --- Comment #19 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Sorry, reverted - tests were failing. perl t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t not ok 9 - Exception thrown on invalid value in the marclist param # Failed test 'Exception thrown on invalid value in the marclist param' # at t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t line 53. # expecting: Koha::Exceptions::WrongParameter # found: Can't locate object method "throw" via package "Koha::Exceptions::WrongParameter" (perhaps you forgot to load "Koha::Exceptions::WrongParameter"?) at /home/vagrant/kohaclone/Koha/SearchEngine/Elasticsearch/QueryBuilder.pm line 432. 1..9 # Looks like you failed 1 test of 9. not ok 1 - build_authorities_query_compat() tests # Failed test 'build_authorities_query_compat() tests' # at t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t line 55. # Looks like you failed 1 test of 1. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18624 Mason James <mtj@kohaaloha.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mtj@kohaaloha.com --- Comment #20 from Mason James <mtj@kohaaloha.com> --- (In reply to Katrin Fischer from comment #19)
Sorry, reverted - tests were failing.
Skipping for 16.05.x, due to failing tests -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18624 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to Stable |RESOLVED Resolution|--- |FIXED --- Comment #21 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- There are too many ElasticSearch patches missing from 16.11.x - better to focus on the more current vesions. Closing. -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org