[Bug 29632] New: Callnumber sorting is incorrect in Elasticsearch
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29632 Bug ID: 29632 Summary: Callnumber sorting is incorrect in Elasticsearch Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Searching - Elasticsearch Assignee: koha-bugs@lists.koha-community.org Reporter: nick@bywatersolutions.com It looks like we are not sorting cn_sort as a raw field, but sorting the analyzed version. To recreate: 0 - Be using ES with Koha 1 - On records with single item, add callnumbers: VA65 E7 R63 1984 VA65 E7 T35 1990 VA65 E45 R67 1985 2 - Add public note 'shrimp' or something to make them easily searchable as a group 3 - Search for 'shrimp' 4 - Note E45 comes last, it should come first Problem may be here: 1122 sub _sort_field { 1123 my ($self, $f) = @_; 1124 1125 my $mappings = $self->get_elasticsearch_mappings(); 1126 my $textField = defined $mappings->{data}{properties}{$f}{type} && $mappings->{data}{properties}{$f}{type} eq 'text'; 1127 if (!defined $self->sort_fields()->{$f} || $self->sort_fields()->{$f}) { 1128 $f .= '__sort'; 1129 } else { 1130 # We need to add '.raw' to text fields without a sort field, 1131 # otherwise it'll sort based on the tokenised form. 1132 $f .= '.raw' if $textField; 1133 } 1134 return $f; 1135 } We need to enforce raw for cn-sort, or we need to index it unanalyzed -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29632 Erica Rohlfs <erica.rohlfs@equinoxOLI.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |erica.rohlfs@equinoxOLI.org --- Comment #1 from Erica Rohlfs <erica.rohlfs@equinoxOLI.org> --- Noting that I recreated this workflow on Koha version 20.11. My results returned step 4 as the second result. So, in my testing, Koha returned results in the following order: VA65 E7 T35 1990, VA65 E45 R67 1985, VA65 E7 R63 1984. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29632 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m Depends on| |26051 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26051 [Bug 26051] Elasticsearch uses the wrong field for callnumber sorting -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29632 Heather <heather_hernandez@nps.gov> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |heather_hernandez@nps.gov -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29632 --- Comment #2 from Heather <heather_hernandez@nps.gov> --- We're having this problem on 21.05 and can be seen in our OPAC: https://keys.bywatersolutions.com/cgi-bin/koha/opac-search.pl?q=callnum%2Cphr%3Ava65&offset=80&sort_by=call_number_asc You'll see that VA65 E45 is still sorting *after* VA65 E7. It's a big problem for us to continue to have the LC call numbers not sorted correctly--the staff can't sort correctly for paging lists, for example, and being a closed stack library, this impacts us daily, with every service request, where they're operating from Carts, searches, lists, and often generating their own paging/pull lists from various parts of Koha. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29632 koha-US bug tracker <bugzilla@koha-us.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bugzilla@koha-us.org -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29632 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29632 --- Comment #3 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 136682 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=136682&action=edit Bug 29632: Don't sort cn-sort numerically When defining our sort fields in we defined all as 'numeric' For other string containing numbers this is likely correct, however, for callnumbers it is not. e.g. E45 should sort before E7 This patch adds a new 'callnumber' type and deifnes this for cn-sort and adds to the field maping a sort without numeric set To test: 0 - Be using ES with Koha 1 - On records with single item, add callnumbers: VA65 E7 R63 1984 VA65 E7 T35 1990 VA65 E45 R67 1985 2 - Add public note 'shrimp' or something to make them easily searchable as a group 3 - Search for 'shrimp', sort by callnumber 4 - Note E45 comes last, it should come first 5 - Apply patch 6 - Reset ES mappings 7 - Reindex ES 8 - Repeat search 9 - Sorting should be correct when set to callnumber -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29632 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29632 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #136682|0 |1 is obsolete| | --- Comment #4 from David Nind <david@davidnind.com> --- Created attachment 136754 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=136754&action=edit Bug 29632: Don't sort cn-sort numerically When defining our sort fields in we defined all as 'numeric' For other string containing numbers this is likely correct, however, for callnumbers it is not. e.g. E45 should sort before E7 This patch adds a new 'callnumber' type and deifnes this for cn-sort and adds to the field maping a sort without numeric set To test: 0 - Be using ES with Koha 1 - On records with single item, add callnumbers: VA65 E7 R63 1984 VA65 E7 T35 1990 VA65 E45 R67 1985 2 - Add public note 'shrimp' or something to make them easily searchable as a group 3 - Search for 'shrimp', sort by callnumber 4 - Note E45 comes last, it should come first 5 - Apply patch 6 - Reset ES mappings 7 - Reindex ES 8 - Repeat search 9 - Sorting should be correct when set to callnumber Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29632 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david@davidnind.com --- Comment #5 from David Nind <david@davidnind.com> --- Testing notes (using koha-testing-docker): - After applying the patch (step 5) you need to run a database update (updatedatabase0 -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29632 Michal Urban <michalurban177@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #136754|0 |1 is obsolete| | --- Comment #6 from Michal Urban <michalurban177@gmail.com> --- Created attachment 137182 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=137182&action=edit Bug 29632: Don't sort cn-sort numerically When defining our sort fields in we defined all as 'numeric' For other string containing numbers this is likely correct, however, for callnumbers it is not. e.g. E45 should sort before E7 This patch adds a new 'callnumber' type and deifnes this for cn-sort and adds to the field maping a sort without numeric set To test: 0 - Be using ES with Koha 1 - On records with single item, add callnumbers: VA65 E7 R63 1984 VA65 E7 T35 1990 VA65 E45 R67 1985 2 - Add public note 'shrimp' or something to make them easily searchable as a group 3 - Search for 'shrimp', sort by callnumber 4 - Note E45 comes last, it should come first 5 - Apply patch 6 - Reset ES mappings 7 - Reindex ES 8 - Repeat search 9 - Sorting should be correct when set to callnumber Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Michal Urban <michalurban177@gmail.com> -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29632 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |martin.renvoize@ptfs-europe | |.com Status|Signed Off |Failed QA --- Comment #7 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Looks like we're missing a unit test here for the Koha/SearchEngine/Elasticsearch.pm change. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29632 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact| |martin.renvoize@ptfs-europe | |.com -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29632 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |nick@bywatersolutions.com |ity.org | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29632 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29632 --- Comment #8 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 137539 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=137539&action=edit Bug 29632: Unit tests This patch adds unit tests, as well as changing existing test to use a mock and read the data as passed in tests, rather than relying on what exists in the DB -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29632 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|--- |Small patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29632 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #137182|0 |1 is obsolete| | --- Comment #9 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 137605 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=137605&action=edit Bug 29632: Don't sort cn-sort numerically When defining our sort fields in we defined all as 'numeric' For other string containing numbers this is likely correct, however, for callnumbers it is not. e.g. E45 should sort before E7 This patch adds a new 'callnumber' type and deifnes this for cn-sort and adds to the field maping a sort without numeric set To test: 0 - Be using ES with Koha 1 - On records with single item, add callnumbers: VA65 E7 R63 1984 VA65 E7 T35 1990 VA65 E45 R67 1985 2 - Add public note 'shrimp' or something to make them easily searchable as a group 3 - Search for 'shrimp', sort by callnumber 4 - Note E45 comes last, it should come first 5 - Apply patch 6 - Reset ES mappings 7 - Reindex ES 8 - Repeat search 9 - Sorting should be correct when set to callnumber Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Michal Urban <michalurban177@gmail.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29632 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #137539|0 |1 is obsolete| | --- Comment #10 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 137606 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=137606&action=edit Bug 29632: Unit tests This patch adds unit tests, as well as changing existing test to use a mock and read the data as passed in tests, rather than relying on what exists in the DB Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29632 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA --- Comment #11 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Thanks Nick, Passing QA now :) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29632 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the| |This fixes the sorting of release notes| |call numbers when using | |Elasticsearch. Sorting will | |now work correctly for | |non-numeric call numbers, | |for example, E45 will now | |sort before E7. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29632 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |tomascohen@gmail.com --- Comment #12 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Shouldn't we add the index to the configuration on the update? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29632 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to master Version(s)| |22.11.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29632 --- Comment #13 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Pushed to master for 22.11. Nice work everyone, thanks! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29632 --- Comment #14 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 137823 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=137823&action=edit Bug 29632: (QA follow-up) Add ENUM value to kohastructure.sql :-D Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29632 --- Comment #15 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 138611 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=138611&action=edit Bug 29632: (QA follow-up) Fix COMMENT discrepancy on upgrade Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29632 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|22.11.00 |22.11.00, 22.05.05 released in| | Status|Pushed to master |Pushed to stable CC| |lucas@bywatersolutions.com --- Comment #16 from Lucas Gass <lucas@bywatersolutions.com> --- Backported to 22.05.x for 22.05.05 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29632 Arthur Suzuki <arthur.suzuki@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED CC| |arthur.suzuki@biblibre.com Status|Pushed to stable |RESOLVED --- Comment #17 from Arthur Suzuki <arthur.suzuki@biblibre.com> --- conflicts in templates when trying to apply on 21.11. Provide a backport if needed. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29632 --- Comment #18 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 139827 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=139827&action=edit Bug 29632: [21.11.X] Don't sort cn-sort numerically When defining our sort fields in we defined all as 'numeric' For other string containing numbers this is likely correct, however, for callnumbers it is not. e.g. E45 should sort before E7 This patch adds a new 'callnumber' type and deifnes this for cn-sort and adds to the field maping a sort without numeric set To test: 0 - Be using ES with Koha 1 - On records with single item, add callnumbers: VA65 E7 R63 1984 VA65 E7 T35 1990 VA65 E45 R67 1985 2 - Add public note 'shrimp' or something to make them easily searchable as a group 3 - Search for 'shrimp', sort by callnumber 4 - Note E45 comes last, it should come first 5 - Apply patch 6 - Reset ES mappings 7 - Reindex ES 8 - Repeat search 9 - Sorting should be correct when set to callnumber Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Michal Urban <michalurban177@gmail.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Bug 29632: Unit tests This patch adds unit tests, as well as changing existing test to use a mock and read the data as passed in tests, rather than relying on what exists in the DB Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Bug 29632: (QA follow-up) Add ENUM value to kohastructure.sql :-D Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Bug 29632: (QA follow-up) Fix COMMENT discrepancy on upgrade Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29632 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |REOPENED Resolution|FIXED |--- -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29632 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|REOPENED |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29632 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29632 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29632 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to stable -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29632 --- Comment #19 from Nick Clemens <nick@bywatersolutions.com> --- (In reply to Arthur Suzuki from comment #17)
conflicts in templates when trying to apply on 21.11. Provide a backport if needed.
Patch supplied -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org