[Bug 40658] New: When sorting by local-number we should use the sort field
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 Bug ID: 40658 Summary: When sorting by local-number we should use the sort field Change sponsored?: --- Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Searching - Elasticsearch Assignee: koha-bugs@lists.koha-community.org Reporter: nick@bywatersolutions.com QA Contact: testopia@bugs.koha-community.org When adding 'local-number' as a fallback sort I explicitly added sorting on 'local-number' - it should be 'local-number__sort' This avoids problems when mappings may be older and local-number itself is indexed as a string and not a number -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |23875 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 [Bug 23875] Elasticsearch - When sorting by score we should provide a tiebreaker -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 --- Comment #1 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 185441 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=185441&action=edit Bug 40658: Use local-number__sort and not local-number directly This patch makes sure we inject the __sort form of local-number, rather than the field directly - this ensurs searching works even if mappings mayu be different and local-number itself is not numeric To test: 1 - Edit mappings.yaml and set local-number from 'number' to 'string' NOTE: There are two copies of local-number - one in auth and one in biblio - the latter is the important one here 2 - perl misc/search_tools/rebuild_elasticsearch.pl -r -v 3 - Confirm in staff interface Admin -> Search Engine Configuration that local-number is now a string 4 - Search and get an error Error: Unable to perform your search. Please try again. In the logs: [illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [local-number] in order to load field data by uninverting the inverted index. Note that this can use significant memory. 5 - Apply patch, restart all 6 - Search again 7 - Success! -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |nick@bywatersolutions.com |ity.org | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 --- Comment #2 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 185447 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=185447&action=edit Bug 40658: Atomic update to ensure local-number is sortable I don't know if this is appropriate for master as it requires Koha objects, but this is helpful for upgrading sites to avoid any problems from old mappings for local-number -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #185441|0 |1 is obsolete| | --- Comment #3 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 185511 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=185511&action=edit Bug 40658: Use local-number__sort and not local-number directly This patch makes sure we inject the __sort form of local-number, rather than the field directly - this ensurs searching works even if mappings mayu be different and local-number itself is not numeric To test: 1 - Edit mappings.yaml and set local-number from 'number' to 'string' NOTE: There are two copies of local-number - one in auth and one in biblio - the latter is the important one here 2 - perl misc/search_tools/rebuild_elasticsearch.pl -r -v 3 - Confirm in staff interface Admin -> Search Engine Configuration that local-number is now a string 4 - Search and get an error Error: Unable to perform your search. Please try again. In the logs: [illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [local-number] in order to load field data by uninverting the inverted index. Note that this can use significant memory. 5 - Apply patch, restart all 6 - Search again 7 - Success! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #185447|0 |1 is obsolete| | --- Comment #4 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 185512 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=185512&action=edit Bug 40658: Atomic update to ensure local-number is sortable I don't know if this is appropriate for master as it requires Koha objects, but this is helpful for upgrading sites to avoid any problems from old mappings for local-number -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 Brendan Lawlor <blawlor@clamsnet.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |blawlor@clamsnet.org --- Comment #5 from Brendan Lawlor <blawlor@clamsnet.org> --- I tried to test this, up until step 4 went as planned, but I couldn't get the search to work after applying the patch. I started ktd with ktd --es7 up I got: No results found You did not specify any search criteria. Error: Unable to perform your search. Please try again. Maybe I'm missing some system preference or process? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 Andrii Nugged <nugged@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nugged@gmail.com --- Comment #6 from Andrii Nugged <nugged@gmail.com> --- Is atomic update ElasticSearch hardcoded? What if in case when we have Zebra? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 --- Comment #7 from Andrii Nugged <nugged@gmail.com> --- (In reply to Nick Clemens (kidclamp) from comment #4)
Created attachment 185512 [details] [review] Bug 40658: Atomic update to ensure local-number is sortable
Is atomic update ElasticSearch hardcoded? What if in case when we have Zebra? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 Bernard <bernard.scaife@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #185511|0 |1 is obsolete| | --- Comment #8 from Bernard <bernard.scaife@openfifth.co.uk> --- Created attachment 189385 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=189385&action=edit Bug 40658: Use local-number__sort and not local-number directly This patch makes sure we inject the __sort form of local-number, rather than the field directly - this ensurs searching works even if mappings mayu be different and local-number itself is not numeric To test: 1 - Edit mappings.yaml and set local-number from 'number' to 'string' NOTE: There are two copies of local-number - one in auth and one in biblio - the latter is the important one here 2 - perl misc/search_tools/rebuild_elasticsearch.pl -r -v 3 - Confirm in staff interface Admin -> Search Engine Configuration that local-number is now a string 4 - Search and get an error Error: Unable to perform your search. Please try again. In the logs: [illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [local-number] in order to load field data by uninverting the inverted index. Note that this can use significant memory. 5 - Apply patch, restart all 6 - Search again 7 - Success! Signed-off-by: Bernard Scaife <bernard.scaife@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david@davidnind.com --- Comment #9 from David Nind <david@davidnind.com> --- Hi Bernard. It looks like you signed this off in November. Could you change the status to Signed Off? That way, you will also get credit on the dashboard (https://dashboard.koha-community.org/leaderboard). The patch still applies. Thanks! David -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m --- Comment #10 from Fridolin Somers <fridolin.somers@biblibre.com> --- Ah fist patch is missing signed-off line -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |tomascohen@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 --- Comment #11 from David Nind <david@davidnind.com> --- I thought I'd attempt this and sign off so it gets to the QA stage, however I'm at a loss for how to do step 4 and generate an error. Is this a normal search, or something else? An example would be helpful. In addition, is this Elasticsearch only, or should this be tested using Zebra as well? Testing notes so far (using KTD): 1. Step 1. Edit admin/searchengine/elasticsearch/mappings.yaml 2. Step 3. Administration > Catalog > Search engine configuration (Elasticsearch) > Search fields tab 3. Step 4. No idea how to do this. A normal search doesn't return any errors. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 --- Comment #12 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- (In reply to David Nind from comment #11)
I thought I'd attempt this and sign off so it gets to the QA stage, however I'm at a loss for how to do step 4 and generate an error.
Is this a normal search, or something else? An example would be helpful.
In addition, is this Elasticsearch only, or should this be tested using Zebra as well?
Testing notes so far (using KTD): 1. Step 1. Edit admin/searchengine/elasticsearch/mappings.yaml 2. Step 3. Administration > Catalog > Search engine configuration (Elasticsearch) > Search fields tab 3. Step 4. No idea how to do this. A normal search doesn't return any errors.
It only needs to be in elasticsearch - the code for zebra doesn't have the same issue Once the settings are changed and reindexed, any search should trigger the error, or any search that has the fallback, I search for 'a' -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #185512|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #189385|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 --- Comment #13 from David Nind <david@davidnind.com> --- Created attachment 195862 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=195862&action=edit Bug 40658: Atomic update to ensure local-number is sortable I don't know if this is appropriate for master as it requires Koha objects, but this is helpful for upgrading sites to avoid any problems from old mappings for local-number Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 --- Comment #14 from David Nind <david@davidnind.com> --- Created attachment 195863 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=195863&action=edit Bug 40658: Use local-number__sort and not local-number directly This patch makes sure we inject the __sort form of local-number, rather than the field directly - this ensurs searching works even if mappings mayu be different and local-number itself is not numeric To test: 1 - Edit mappings.yaml and set local-number from 'number' to 'string' NOTE: There are two copies of local-number - one in auth and one in biblio - the latter is the important one here 2 - perl misc/search_tools/rebuild_elasticsearch.pl -r -v 3 - Confirm in staff interface Admin -> Search Engine Configuration that local-number is now a string 4 - Search and get an error Error: Unable to perform your search. Please try again. In the logs: [illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [local-number] in order to load field data by uninverting the inverted index. Note that this can use significant memory. 5 - Apply patch, restart all 6 - Search again 7 - Success! Signed-off-by: Bernard Scaife <bernard.scaife@openfifth.co.uk> Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 --- Comment #15 from David Nind <david@davidnind.com> --- (In reply to Nick Clemens (kidclamp) from comment #12)
(In reply to David Nind from comment #11)
I thought I'd attempt this and sign off so it gets to the QA stage, however I'm at a loss for how to do step 4 and generate an error.
Is this a normal search, or something else? An example would be helpful.
In addition, is this Elasticsearch only, or should this be tested using Zebra as well?
Testing notes so far (using KTD): 1. Step 1. Edit admin/searchengine/elasticsearch/mappings.yaml 2. Step 3. Administration > Catalog > Search engine configuration (Elasticsearch) > Search fields tab 3. Step 4. No idea how to do this. A normal search doesn't return any errors.
It only needs to be in elasticsearch - the code for zebra doesn't have the same issue
Once the settings are changed and reindexed, any search should trigger the error, or any search that has the fallback, I search for 'a'
Thanks Nick, that worked! I've added my sign-off so that it gets in the QA queue. Updated testing notes (using KTD): 1. Step 1. Edit admin/searchengine/elasticsearch/mappings.yaml 2. Step 3. Administration > Catalog > Search engine configuration (Elasticsearch) > Search fields tab 3. Step 4. Search for 'a'. Error message is in /var/log/koha/kohadev/plack-intranet-error.log 4. Before signing off, I needed to reset the changes made to the mappings file (git checkout admin/searchengine/elasticsearch/mappings.yaml) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 Thomas Klausner <domm@plix.at> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |domm@plix.at |y.org | CC| |domm@plix.at -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 Thomas Klausner <domm@plix.at> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA --- Comment #16 from Thomas Klausner <domm@plix.at> --- All works according to test plan! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 Thomas Klausner <domm@plix.at> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #195862|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 Thomas Klausner <domm@plix.at> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #195863|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 --- Comment #17 from Thomas Klausner <domm@plix.at> --- Created attachment 198382 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=198382&action=edit Bug 40658: Atomic update to ensure local-number is sortable I don't know if this is appropriate for master as it requires Koha objects, but this is helpful for upgrading sites to avoid any problems from old mappings for local-number Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Thomas Klausner <domm@plix.at> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 --- Comment #18 from Thomas Klausner <domm@plix.at> --- Created attachment 198383 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=198383&action=edit Bug 40658: Use local-number__sort and not local-number directly This patch makes sure we inject the __sort form of local-number, rather than the field directly - this ensurs searching works even if mappings mayu be different and local-number itself is not numeric To test: 1 - Edit mappings.yaml and set local-number from 'number' to 'string' NOTE: There are two copies of local-number - one in auth and one in biblio - the latter is the important one here 2 - perl misc/search_tools/rebuild_elasticsearch.pl -r -v 3 - Confirm in staff interface Admin -> Search Engine Configuration that local-number is now a string 4 - Search and get an error Error: Unable to perform your search. Please try again. In the logs: [illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [local-number] in order to load field data by uninverting the inverted index. Note that this can use significant memory. 5 - Apply patch, restart all 6 - Search again 7 - Success! Signed-off-by: Bernard Scaife <bernard.scaife@openfifth.co.uk> Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Thomas Klausner <domm@plix.at> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 Lucas Gass (lukeg) <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to main Version(s)| |26.05.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 --- Comment #19 from Lucas Gass (lukeg) <lucas@bywatersolutions.com> --- Nice work everyone! Pushed to main for 26.05 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 Lucas Gass (lukeg) <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lucas@bywatersolutions.com Keywords| |release-notes-needed -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |rel_25_11_candidate -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 --- Comment #20 from Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> --- Please backport :-D -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 Jacob O'Mara <jacob.omara@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to main |Pushed to stable Version(s)|26.05.00 |26.05.00,25.11.05 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 --- Comment #21 from Jacob O'Mara <jacob.omara@openfifth.co.uk> --- Thanks all, pushed to 25.11.x -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |42795 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42795 [Bug 42795] Bug 40658 breaks CLI for 25.11 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |43231 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43231 [Bug 43231] Bug 40658 breaks CLI for 25.11 -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org