[Bug 27070] New: Elasticsearch - with Elasticsearch 6 searches failing unless all terms are in the same field
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27070 Bug ID: 27070 Summary: Elasticsearch - with Elasticsearch 6 searches failing unless all terms are in the same field Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: major Priority: P5 - low Component: Searching - Elasticsearch Assignee: koha-bugs@lists.koha-community.org Reporter: nick@bywatersolutions.com To recreate - be using Elasticsearch 6 in KOha On sample data try searching for: street shuffle cool, that works Now try searching for a word from title and the publisher name: shuffle constable No results, odd. Try searching for: shuffle AND constable That works. I cannot figure from the docs why this is the case - we pass a default operator, that should be enough I thought, but no. https://www.elastic.co/guide/en/elasticsearch/reference/6.8/query-dsl-query-... for comparison: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/query-dsl-query-... After testing the type => 'cross_fields' seems to do what we want -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27070 --- Comment #1 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 113902 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=113902&action=edit Bug 27070: Add cross_fields type to our searches -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27070 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #113902|0 |1 is obsolete| | --- Comment #2 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 113903 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=113903&action=edit Bug 27070: Add cross_fields type to our searches This patch adds the 'cross_fields' type to our searches: https://www.elastic.co/guide/en/elasticsearch/reference/6.8/query-dsl-query-... Without this patch the search terms seem to all require being in the same field when using Elasticsearch 6 To test: 1 - Find a record with a title and publisher 2 - Search for a word form the title and confirm the record is returned 3 - Search for a work from the title and the publisher's name 4 - The record is not returned 5 - Apply patch 6 - Repeat #3 7 - The record is returned -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27070 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |andrew@bywatersolutions.com | |, | |fridolin.somers@biblibre.co | |m, | |jonathan.druart@bugs.koha-c | |ommunity.org, | |julian.maurice@biblibre.com | |, | |martin.renvoize@ptfs-europe | |.com, | |mkstephens@lancasterseminar | |y.edu, | |severine.queune@bulac.fr, | |tomascohen@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27070 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff Assignee|koha-bugs@lists.koha-commun |nick@bywatersolutions.com |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27070 Didier Gautheron <didier.gautheron@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA CC| |didier.gautheron@biblibre.c | |om --- Comment #3 from Didier Gautheron <didier.gautheron@biblibre.com> --- Hi It seems diff --git a/C4/Circulation.pm b/C4/Circulation.pm is an unrelated leftover, it adds an unused reasons hash. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27070 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27070 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #113903|0 |1 is obsolete| | --- Comment #4 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 113907 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=113907&action=edit Bug 27070: Add cross_fields type to our searches This patch adds the 'cross_fields' type to our searches: https://www.elastic.co/guide/en/elasticsearch/reference/6.8/query-dsl-query-... Without this patch the search terms seem to all require being in the same field when using Elasticsearch 6 To test: 1 - Find a record with a title and publisher 2 - Search for a word form the title and confirm the record is returned 3 - Search for a work from the title and the publisher's name 4 - The record is not returned 5 - Apply patch 6 - Repeat #3 7 - The record is returned -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27070 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #113907|0 |1 is obsolete| | --- Comment #5 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 113908 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=113908&action=edit Bug 27070: Add cross_fields type to our searches This patch adds the 'cross_fields' type to our searches: https://www.elastic.co/guide/en/elasticsearch/reference/6.8/query-dsl-query-... Without this patch the search terms seem to all require being in the same field when using Elasticsearch 6 To test: 0 - Set QueryAutoTruncate to 'only if * is added' 1 - Find a record with a title and publisher 2 - Search for a word form the title and confirm the record is returned 3 - Search for a work from the title and the publisher's name 4 - The record is not returned 5 - Apply patch 6 - Repeat #3 7 - The record is returned -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27070 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #113908|0 |1 is obsolete| | --- Comment #6 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Created attachment 113911 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=113911&action=edit Bug 27070: Add cross_fields type to our searches This patch adds the 'cross_fields' type to our searches: https://www.elastic.co/guide/en/elasticsearch/reference/6.8/query-dsl-query-... Without this patch the search terms seem to all require being in the same field when using Elasticsearch 6 To test: 0 - Set QueryAutoTruncate to 'only if * is added' 1 - Find a record with a title and publisher 2 - Search for a word form the title and confirm the record is returned 3 - Search for a work from the title and the publisher's name 4 - The record is not returned 5 - Apply patch 6 - Repeat #3 7 - The record is returned Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27070 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off CC| |victor@tuxayo.net -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27070 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |rel_20_11_target -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27070 Joonas Kylmälä <joonas.kylmala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |joonas.kylmala@helsinki.fi --- Comment #7 from Joonas Kylmälä <joonas.kylmala@helsinki.fi> --- Any idea how well this works with weighted search fields? There was some discussion in https://discuss.elastic.co/t/cross-fields-and-boost-is-it-feasible/134219/9 about it and also https://www.elastic.co/guide/en/elasticsearch/reference/master/query-dsl-mul... mentions it but I don't see any immediate issues in our use case. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27070 --- Comment #8 from Fridolin Somers <fridolin.somers@biblibre.com> --- Whooo I was looking for this fix since weeks, thanks a lot -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27070 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |alex.arnaud@biblibre.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27070 --- Comment #9 from Nick Clemens <nick@bywatersolutions.com> --- (In reply to Joonas Kylmälä from comment #7)
Any idea how well this works with weighted search fields? There was some discussion in https://discuss.elastic.co/t/cross-fields-and-boost-is-it-feasible/134219/9 about it and also https://www.elastic.co/guide/en/elasticsearch/reference/master/query-dsl- multi-match-query.html#type-cross-fields mentions it but I don't see any immediate issues in our use case.
It seems mostly that they are saying cross_fields does more than just searching all fields, it tries to guess which fields are most relevant to each term and boost them - so adding field boosts on top of that kind of ruins the work. The problem I see is that the multi field query says it should look for each term in each field with 'OR', i.e. (content:this OR name:this) AND (content:that OR name:that) but what is returned actually seems to be: (content:this AND that) AND (name:this AND that) possibly the problem is with our 'default_operator' setting of 'AND' I think with OR we might end up with results that don't contain all terms, but the ones that do should be most relevant, but our field boosting may push partial matches to the top. cross_fields seems to at least generate the search we expect right now. I am open to discussion, but the current state here is problematic for many users -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27070 Joonas Kylmälä <joonas.kylmala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27070 Joonas Kylmälä <joonas.kylmala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #113911|0 |1 is obsolete| | --- Comment #10 from Joonas Kylmälä <joonas.kylmala@helsinki.fi> --- Created attachment 113976 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=113976&action=edit Bug 27070: Add cross_fields type to our searches This patch adds the 'cross_fields' type to our searches: https://www.elastic.co/guide/en/elasticsearch/reference/6.8/query-dsl-query-... Without this patch the search terms seem to all require being in the same field when using Elasticsearch 6 To test: 0 - Set QueryAutoTruncate to 'only if * is added' 1 - Find a record with a title and publisher 2 - Search for a word form the title and confirm the record is returned 3 - Search for a work from the title and the publisher's name 4 - The record is not returned 5 - Apply patch 6 - Repeat #3 7 - The record is returned Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27070 --- Comment #11 from Joonas Kylmälä <joonas.kylmala@helsinki.fi> --- Using cross_fields here makes this now so that it doesn't try to do an AND search on one field only for all the search keywords which is pretty much guaranteed to fail as people here noticed. The searches now at least bring all the results that contain those keywords, when previously very much relevant results were missing! I'm passing QA for this given it is a big improvement to the previous state but in the future we still need to verify and read the Elasticsearch docs if there could be a even better way doing searches. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27070 --- Comment #12 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Joonas, could you open a follow-up bug report with the links/ideas to explore? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27070 Joonas Kylmälä <joonas.kylmala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=27091 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27070 --- Comment #13 from Joonas Kylmälä <joonas.kylmala@helsinki.fi> --- (In reply to Jonathan Druart from comment #12)
Joonas, could you open a follow-up bug report with the links/ideas to explore?
I opened bug 27091. It's gonna be tedious process but I'm actually planning to start doing in the following 3 months, hopefully also being ready then :) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27070 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |20.11.00 released in| | Status|Passed QA |Pushed to master -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27070 --- Comment #14 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Pushed to master for 20.11, thanks to everybody involved! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27070 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|rel_20_11_target | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27070 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to master |Pushed to stable Version(s)|20.11.00 |20.11.00, 20.05.07 released in| | --- Comment #15 from Andrew Fuerste-Henry <andrew@bywatersolutions.com> --- Pushed to 20.05.x for 20.05.07 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27070 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to stable |Pushed to oldstable -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27070 --- Comment #16 from Nick Clemens <nick@bywatersolutions.com> --- There is an issue here, this breaks searching with Elasticsearch 5.x It may be okay, ES5 is no longer supported, so this may be the point at which we want to support only Es6+, but we will need to announce that -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27070 --- Comment #17 from Victor Grousset/tuxayo <victor@tuxayo.net> --- EOL since 2019-03-11 IIUC https://www.elastic.co/support/eol -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27070 --- Comment #18 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to Nick Clemens from comment #16)
There is an issue here, this breaks searching with Elasticsearch 5.x
It may be okay, ES5 is no longer supported, so this may be the point at which we want to support only Es6+, but we will need to announce that
In that case I am suggesting to remove it from 20.05, before .7 is released. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27070 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |27252 --- Comment #19 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- And add a warning on the about page, see bug 27252. Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27252 [Bug 27252] ES5 no longer supported (since 20.11.00) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27070 --- Comment #20 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Does this mean that for now, 19.11 and 20.05 will have to advertise (and test in the CI) ES 5 only? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27070 --- Comment #21 from Victor Grousset/tuxayo <victor@tuxayo.net> ---
Does this mean that for now, 19.11 and 20.05 will have to advertise (and test in the CI) ES 5 only?
Nope, they can support ES 6 and ES 5 thanks to bug 27252 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27070 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|20.11.00, 20.05.07 |20.11.00, 20.05.07, released in| |19.11.14 Status|Pushed to oldstable |Pushed to oldoldstable --- Comment #22 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Backported: Pushed to 19.11.x branch for 19.11.14 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27070 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |27448 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27448 [Bug 27448] Update ElasticsearchCrossFields description to explain consequences of setting in Koha -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27070 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks|27448 | Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27448 [Bug 27448] Update ElasticsearchCrossFields description to explain consequences of setting in Koha -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org