[Bug 23089] New: Elasticsearch - cannot sort on non-text fields
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 Bug ID: 23089 Summary: Elasticsearch - cannot sort on non-text fields Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Searching - Elasticsearch Assignee: koha-bugs@lists.koha-community.org Reporter: nick@bywatersolutions.com Target Milestone: --- issues is defined as an sum field, this is interpreted to ES as integer, however, the issues__sort field is stored as keyword. Attempting to order search results by 'popularity' fails from ES - because we can't use text fields, we have to use a 'keyword' field. In our biblio search indexes this is fine because our __sort fields all get a phrase keyword field We onyl set field__sort.phrase for text fields though, so issues__sort doesn't use the phrase field, even though it exists Adding a test to try to ilustrate what I am saying here To recreate: 1 - Enable and index ES 2 - Search on staff client or OPAC 3 - Attempt to sort by popularity 4 - It fails, check the error logs to see why -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |alex.arnaud@biblibre.com, | |ere.maijala@helsinki.fi, | |glasklas@gmail.com, | |mkstephens@lancasterseminar | |y.edu, | |nicolas.legrand@bulac.fr -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 --- Comment #1 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 90451 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=90451&action=edit Bug 23089: Unit test to highlight sorting issue -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |ASSIGNED -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 e.betemps <e.betemps@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |e.betemps@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Needs Signoff Patch complexity|--- |Trivial patch -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 --- Comment #2 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 93982 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=93982&action=edit Bug 23089: Unit tests -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 --- Comment #3 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 93983 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=93983&action=edit Bug 23089: Fix sorting on sum and regular fields As of bug 20589 we no longer analyze sort fields and so we no longer need to append ".phrase" to our sort in searches. Additionally, sort fields based on 'sum' should also use sum in building the value to sort on To test: 0 - Be using ES 1 - Find the most circulated item in your collection 2 - Search for '*' 3 - Sort by popularity DESC 4 - Note that item is not first 5 - Try to sort by anything but relevancy, it fails 6 - Apply patch 7 - Redo searches and sorts 8 - Things should now work as expected -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #90451|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |23875 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 [Bug 23875] Elasticsearch - ES - When sorting by score we should provide a tiebreaker -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|normal |critical Assignee|koha-bugs@lists.koha-commun |nick@bywatersolutions.com |ity.org | --- Comment #4 from Nick Clemens <nick@bywatersolutions.com> --- Bumping relevancy, we must fix sorting before release -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 Ere Maijala <ere.maijala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 Ere Maijala <ere.maijala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #93982|0 |1 is obsolete| | --- Comment #5 from Ere Maijala <ere.maijala@helsinki.fi> --- Created attachment 94617 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=94617&action=edit Bug 23089: Unit tests Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 Ere Maijala <ere.maijala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #93983|0 |1 is obsolete| | --- Comment #6 from Ere Maijala <ere.maijala@helsinki.fi> --- Created attachment 94618 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=94618&action=edit Bug 23089: Fix sorting on sum and regular fields As of bug 20589 we no longer analyze sort fields and so we no longer need to append ".phrase" to our sort in searches. Additionally, sort fields based on 'sum' should also use sum in building the value to sort on To test: 0 - Be using ES 1 - Find the most circulated item in your collection 2 - Search for '*' 3 - Sort by popularity DESC 4 - Note that item is not first 5 - Try to sort by anything but relevancy, it fails 6 - Apply patch 7 - Redo searches and sorts 8 - Things should now work as expected Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 Ere Maijala <ere.maijala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 --- Comment #7 from Ere Maijala <ere.maijala@helsinki.fi> --- Created attachment 94653 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=94653&action=edit Bug 23089: Fix QueryBuilder tests. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 --- Comment #8 from Ere Maijala <ere.maijala@helsinki.fi> --- I noticed that QueryBuilder tests were failing since they expected the __phrase suffix. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 Ere Maijala <ere.maijala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |23887 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23887 [Bug 23887] Add support for client-specified sort order in Z3950Responder -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #94653|0 |1 is obsolete| | --- Comment #9 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 94656 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=94656&action=edit Bug 23089: Fix QueryBuilder tests. Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 Séverine Queune <severine.queune@bulac.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |severine.queune@bulac.fr -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 --- Comment #10 from Nick Clemens <nick@bywatersolutions.com> --- These patches address issue on master, we will still need a patch for popularity sort in 19.05 and earlier -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 AspenCat Team <aspencatteam@clicweb.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |aspencatteam@clicweb.org -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #94617|0 |1 is obsolete| | --- Comment #11 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 94944 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=94944&action=edit Bug 23089: Unit tests Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #94618|0 |1 is obsolete| | --- Comment #12 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 94945 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=94945&action=edit Bug 23089: Fix sorting on sum and regular fields As of bug 20589 we no longer analyze sort fields and so we no longer need to append ".phrase" to our sort in searches. Additionally, sort fields based on 'sum' should also use sum in building the value to sort on To test: 0 - Be using ES 1 - Find the most circulated item in your collection 2 - Search for '*' 3 - Sort by popularity DESC 4 - Note that item is not first 5 - Try to sort by anything but relevancy, it fails 6 - Apply patch 7 - Redo searches and sorts 8 - Things should now work as expected Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #94656|0 |1 is obsolete| | --- Comment #13 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 94946 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=94946&action=edit Bug 23089: Fix QueryBuilder tests. Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA CC| |martin.renvoize@ptfs-europe | |.com --- Comment #14 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Works as expected, has tests, passes tests and passes QA script. Passing QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |19.11.00 released in| | Status|Passed QA |Pushed to master -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 --- Comment #15 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Nice work! Pushed to master for 19.11.00 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 Fridolin SOMERS <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m --- Comment #16 from Fridolin SOMERS <fridolin.somers@biblibre.com> --- Can it be applyed on 19.05.x even without Bug 20589 that added changed admin/searchengine/elasticsearch/field_config.yaml -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 --- Comment #17 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 95390 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=95390&action=edit Bug 23089: [19.05 and earlier] Use integer field directly for sorting -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 --- Comment #18 from Nick Clemens <nick@bywatersolutions.com> --- (In reply to Fridolin SOMERS from comment #16)
Can it be applyed on 19.05.x even without Bug 20589 that added changed admin/searchengine/elasticsearch/field_config.yaml
No, it relies on other logic there, but I made a new patch to bridge the behavior. I tried to keep it as small as possible. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |rel_19_05_candidate -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@bugs.koha-c | |ommunity.org --- Comment #19 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Nick, tests are failing on 18.11.x t/Koha/SearchEngine/Elasticsearch.t .. 3/4 # Looks like you planned 51 tests but ran 10. # Failed test 'Koha::SearchEngine::Elasticsearch::marc_records_to_documents () tests' # at t/Koha/SearchEngine/Elasticsearch.t line 518. Not a HASH reference at t/Koha/SearchEngine/Elasticsearch.t line 335. # Looks like your test exited with 255 just after 4. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 --- Comment #20 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- This patchset has been pushed to 19.05.05 and 18.11.11 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lucas@bywatersolutions.com Status|Pushed to master |Pushed to stable --- Comment #21 from Lucas Gass <lucas@bywatersolutions.com> --- This is not in 19.05.05. It has been backported to 19.05.x for 19.05.06 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to stable |Pushed to oldstable -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 --- Comment #22 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 96423 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=96423&action=edit Bug 23089: [18.11.x] fix tests -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 Hayley Mapley <hayleymapley@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |hayleymapley@catalyst.net.n | |z Version(s)|19.11.00 |19.11.00, 18.11.12 released in| | Status|Pushed to oldstable |Pushed to oldoldstable --- Comment #23 from Hayley Mapley <hayleymapley@catalyst.net.nz> --- This patchset has not been pushed to 18.11.11, rather it has been pushed to 18.11.x for 18.11.12 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 Hayley Mapley <hayleymapley@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|19.11.00, 18.11.12 |19.11.00, 19.05.06, released in| |18.11.12 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 --- Comment #24 from Nick Clemens <nick@bywatersolutions.com> --- Note that sites receiving this patch will need to rebuild their ES indexes due to the change to field_config -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org