[Bug 23875] New: Elasticsearch - ES - When sorting by score we should provide a tiebreaker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 Bug ID: 23875 Summary: Elasticsearch - ES - When sorting by score we should provide a tiebreaker Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Searching - Elasticsearch Assignee: koha-bugs@lists.koha-community.org Reporter: nick@bywatersolutions.com In ES if we sort by 'relevance' we actually don't pass anything to ES and results are ordered by score. When doing a search limiting only by collection or type, the scores are all the same. This is fine, except that editing any record will then cause a reordering of the results, even though it won't affect the score. We should provide a tie-breaker in the case of equal sort scores -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|--- |Trivial patch Status|NEW |Needs Signoff -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 --- Comment #1 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 94579 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=94579&action=edit Bug 23875: Explicitly srt searches by score and provide local-number as tie breaker To test: 1 - Set search engine to ES 2 - Search the catalog for "*" 3 - Note results 4 - Edit first result 5 - Repeat search, results are reordered 6 - Apply patch 7 - Search for "*" 8 - Highest biblionumbers are returned first 9 - Edit first result, repeat search 10 - Order does not change 11 - Confirm sorting by author, title, etc passes the same trials -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |23089 Assignee|koha-bugs@lists.koha-commun |nick@bywatersolutions.com |ity.org | CC| |alex.arnaud@biblibre.com, | |ere.maijala@helsinki.fi, | |glasklas@gmail.com, | |jonathan.druart@bugs.koha-c | |ommunity.org Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23089 [Bug 23089] Elasticsearch - cannot sort on non-text fields -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 --- Comment #2 from Ere Maijala <ere.maijala@helsinki.fi> --- Should we care that the sorting by local-number.raw isn't numeric? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 --- Comment #3 from Nick Clemens <nick@bywatersolutions.com> --- (In reply to Ere Maijala from comment #2)
Should we care that the sorting by local-number.raw isn't numeric?
In the mappings local-number is set as a string by default - should we change it to a number? I think in practice it will be a number, however, under ES at least I don't see why someone couldn't map local-number to 001 and manage their own control numbers with letters, so maybe string is okay as long as your numbers are built consistently? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 Ere Maijala <ere.maijala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 Ere Maijala <ere.maijala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #94579|0 |1 is obsolete| | --- Comment #4 from Ere Maijala <ere.maijala@helsinki.fi> --- Created attachment 94712 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=94712&action=edit Bug 23875: Explicitly srt searches by score and provide local-number as tie breaker To test: 1 - Set search engine to ES 2 - Search the catalog for "*" 3 - Note results 4 - Edit first result 5 - Repeat search, results are reordered 6 - Apply patch 7 - Search for "*" 8 - Highest biblionumbers are returned first 9 - Edit first result, repeat search 10 - Order does not change 11 - Confirm sorting by author, title, etc passes the same trials Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 --- Comment #5 from Ere Maijala <ere.maijala@helsinki.fi> --- (In reply to Nick Clemens from comment #3)
In the mappings local-number is set as a string by default - should we change it to a number? I think in practice it will be a number, however, under ES at least I don't see why someone couldn't map local-number to 001 and manage their own control numbers with letters, so maybe string is okay as long as your numbers are built consistently?
Changing it to number would be scary, and I don't think this is really an issue. If we want to make sorting consistent, we could always left-pad the biblionumber with zeros, but I don't really like it since it wouldn't match the biblionumber field in MARC exactly. Alternatively we could have a hardcoded mapping to a numeric biblionumber field. Or make the tie-breaker sort field configurable. But since local-number gives us consistent results, I believe this is good enough for all practical purposes. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |katrin.fischer@bsz-bw.de --- Comment #6 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Ere Maijala from comment #5)
(In reply to Nick Clemens from comment #3)
In the mappings local-number is set as a string by default - should we change it to a number? I think in practice it will be a number, however, under ES at least I don't see why someone couldn't map local-number to 001 and manage their own control numbers with letters, so maybe string is okay as long as your numbers are built consistently?
Changing it to number would be scary, and I don't think this is really an issue. If we want to make sorting consistent, we could always left-pad the biblionumber with zeros, but I don't really like it since it wouldn't match the biblionumber field in MARC exactly. Alternatively we could have a hardcoded mapping to a numeric biblionumber field. Or make the tie-breaker sort field configurable. But since local-number gives us consistent results, I believe this is good enough for all practical purposes.
I am not sure if I understand the issue compeletely, but why would local-number be scary as a number? local-number is usually the biblionumber, right? And if it is, I am in doubt about what Nick says that you could just remap it to 001 - at least I would expect issues with that. I think being able to sort numerical on biblionumber is important so you can use it as a tie breaker as mentioned before, when all other criteria are the some. You you can sort from newer to older. If we don't sort numerical... that meanst 9 999 2 22 1 11 ... something like that? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 --- Comment #7 from Ere Maijala <ere.maijala@helsinki.fi> --- (In reply to Katrin Fischer from comment #6)
I am not sure if I understand the issue compeletely, but why would local-number be scary as a number? local-number is usually the biblionumber, right? And if it is, I am in doubt about what Nick says that you could just remap it to 001 - at least I would expect issues with that.
Yes, it's usually biblionumber, but there's nothing preventing one from e.g. adding other fields in it too. Unless we hard-code a field for biblionumber, we can't positively say it's always a number.
If we don't sort numerical... that meanst 9 999 2 22 1 11 ... something like that?
Yes. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 --- Comment #8 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Ere Maijala from comment #7)
(In reply to Katrin Fischer from comment #6)
I am not sure if I understand the issue compeletely, but why would local-number be scary as a number? local-number is usually the biblionumber, right? And if it is, I am in doubt about what Nick says that you could just remap it to 001 - at least I would expect issues with that.
Yes, it's usually biblionumber, but there's nothing preventing one from e.g. adding other fields in it too. Unless we hard-code a field for biblionumber, we can't positively say it's always a number.
If we don't sort numerical... that meanst 9 999 2 22 1 11 ... something like that?
Yes.
Maybe we should have an unchangeable index for biblionumber then? The sorting could quickly become an issue. People tend to search very 'broadly' and then the tie breaker would come in quickly. At least that's what we see with Zebra quite often with searches for single words and similar. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 --- Comment #9 from Ere Maijala <ere.maijala@helsinki.fi> --- (In reply to Katrin Fischer from comment #8)
Maybe we should have an unchangeable index for biblionumber then? The sorting could quickly become an issue. People tend to search very 'broadly' and then the tie breaker would come in quickly. At least that's what we see with Zebra quite often with searches for single words and similar.
Using non-numeric sort doesn't really matter for tie-breaker unless you care for the order. I'd actually like us to have a hard-coded biblionumber index field. One could also argue that biblionumber should only be the last resort, and publication date should be the primary. We'll probably end up making it configurable. ;) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 --- Comment #10 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- And by the reasoning that we can't prevent other fields being added... doesn't that apply to all other indexes as well? Would Elastic give you an error when searching on a string in a numerical index or just not sort them 'right'?(In reply to Ere Maijala from comment #9)
(In reply to Katrin Fischer from comment #8)
Maybe we should have an unchangeable index for biblionumber then? The sorting could quickly become an issue. People tend to search very 'broadly' and then the tie breaker would come in quickly. At least that's what we see with Zebra quite often with searches for single words and similar.
Using non-numeric sort doesn't really matter for tie-breaker unless you care for the order. I'd actually like us to have a hard-coded biblionumber index field. One could also argue that biblionumber should only be the last resort, and publication date should be the primary. We'll probably end up making it configurable. ;)
publication date can get quickly problematic for serials... the first date in 008 is usually quite old. Every approach has its own problems ;) Maybe we can move the discussion of adding a hardcoded biblionumber index into another bug, but it really bugs me that the sorting of a clearly numerical value would be alphabetic... it just seems very wrong. I've ended up explaining the 'last resort' ordering a lot of times to libraries. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 --- Comment #11 from Ere Maijala <ere.maijala@helsinki.fi> --- (In reply to Katrin Fischer from comment #10)
publication date can get quickly problematic for serials... the first date in 008 is usually quite old. Every approach has its own problems ;)
True, and I didn't mean to imply that should be a default. :)
Maybe we can move the discussion of adding a hardcoded biblionumber index into another bug, but it really bugs me that the sorting of a clearly numerical value would be alphabetic... it just seems very wrong. I've ended up explaining the 'last resort' ordering a lot of times to libraries.
I'm fine either way. But clearly at least a follow-up to this bug is needed. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 Séverine Queune <severine.queune@bulac.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |severine.queune@bulac.fr -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |martin.renvoize@ptfs-europe | |.com --- Comment #12 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Are you intending on continuing here with the requested followup Nick? I think the suggestion was to have a fixed biblionumber index which is numeric and use that for the tie-breaker by default.. as aposed to using the existing 'local-number' index which can be re-mapped? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Patch doesn't apply --- Comment #13 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Patch no longer applies, please rebase! Apply? [(y)es, (n)o, (i)nteractive] y Applying: Bug 23875: Explicitly srt searches by score and provide local-number as tie breaker error: sha1 information is lacking or useless (Koha/SearchEngine/Elasticsearch/QueryBuilder.pm). error: could not build fake ancestor Patch failed at 0001 Bug 23875: Explicitly srt searches by score and provide local-number as tie breaker The copy of the patch that failed is found in: .git/rebase-apply/patch When you have resolved this problem run "git bz apply --continue". If you would prefer to skip this patch, instead run "git bz apply --skip". To restore the original branch and stop patching run "git bz apply --abort". Patch left in /tmp/Bug-23875-Explicitly-srt-searches-by-score-and-pro-n_vYDq.patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #94712|0 |1 is obsolete| | --- Comment #14 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 108391 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=108391&action=edit Bug 23875: Explicitly srt searches by score and provide local-number as tie breaker To test: 1 - Set search engine to ES 2 - Search the catalog for "*" 3 - Note results 4 - Edit first result 5 - Repeat search, results are reordered 6 - Apply patch 7 - Search for "*" 8 - Highest biblionumbers are returned first 9 - Edit first result, repeat search 10 - Order does not change 11 - Confirm sorting by author, title, etc passes the same trials Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Elasticsearch - ES - When |Elasticsearch - When |sorting by score we should |sorting by score we should |provide a tiebreaker |provide a tiebreaker -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Failed QA --- Comment #15 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Martin Renvoize from comment #12)
Are you intending on continuing here with the requested followup Nick?
I think the suggestion was to have a fixed biblionumber index which is numeric and use that for the tie-breaker by default.. as aposed to using the existing 'local-number' index which can be re-mapped?
I really believe we should do that and have a numerical sort on the biblionumber as a tiebreaker. I hear your argument about remapping local-number, but I believe that this is a concern that needs more work and thinking overall if we want to officially support it. At the moment, doing that, would probably break your Zebra pretty badly and now allow to switch back and forth easily anymore. Having a numerical sort on biblionumber on the tiebreaker would also mimick Zebra a bit allowing to compare relevance sort results more easily. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |kyle@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m --- Comment #16 from Fridolin Somers <fridolin.somers@biblibre.com> --- What a strange bug. Any updates ? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=30879 --- Comment #17 from Fridolin Somers <fridolin.somers@biblibre.com> --- 30879 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 --- Comment #18 from Fridolin Somers <fridolin.somers@biblibre.com> --- Ah I had issues with local-number sort, found Bug 30879 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 --- Comment #19 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I believe we have solved the numerical sorting by now - could this be revived? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 --- Comment #20 from Fridolin Somers <fridolin.somers@biblibre.com> --- isnt it possible to sort on '_id' (ES document id) instead of local-number ? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 --- Comment #21 from Fridolin Somers <fridolin.somers@biblibre.com> --- (In reply to Fridolin Somers from comment #20)
isnt it possible to sort on '_id' (ES document id) instead of local-number ?
Ah there seems to be a performance issue with sorting on '_id', bad idea ;) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #108391|0 |1 is obsolete| | --- Comment #22 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 174227 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=174227&action=edit Bug 23875: Explicitly srt searches by score and provide local-number as tie breaker To test: 1 - Set search engine to ES 2 - Search the catalog for "*" 3 - Note results 4 - Edit first result 5 - Repeat search, results are reordered 6 - Apply patch 7 - Search for "*" 8 - Highest biblionumbers are returned first 9 - Edit first result, repeat search 10 - Order does not change 11 - Confirm sorting by author, title, etc passes the same trials -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 --- Comment #23 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- We turned local-number into a number on bug 30879, can we move this one again? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=38741 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Patch doesn't apply CC| |david@davidnind.com --- Comment #24 from David Nind <david@davidnind.com> --- The patch no longer applies 8-(... git bz apply 23875 Bug 23875 - Elasticsearch - When sorting by score we should provide a tiebreaker 174227 - Bug 23875: Explicitly srt searches by score and provide local-number as tie breaker Apply? [(y)es, (n)o, (i)nteractive] y Applying: Bug 23875: Explicitly srt searches by score and provide local-number as tie breaker Using index info to reconstruct a base tree... M Koha/SearchEngine/Elasticsearch/QueryBuilder.pm M t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t Falling back to patching base and 3-way merge... Auto-merging t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t CONFLICT (content): Merge conflict in t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t Auto-merging Koha/SearchEngine/Elasticsearch/QueryBuilder.pm error: Failed to merge in the changes. Patch failed at 0001 Bug 23875: Explicitly srt searches by score and provide local-number as tie breaker -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #174227|0 |1 is obsolete| | --- Comment #25 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 181183 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=181183&action=edit Bug 23875: Explicitly srt searches by score and provide local-number as tie breaker To test: 1 - Set search engine to ES 2 - Search the catalog for "*" 3 - Note results 4 - Edit first result 5 - Repeat search, results are reordered 6 - Apply patch 7 - Search for "*" 8 - Highest biblionumbers are returned first 9 - Edit first result, repeat search 10 - Order does not change 11 - Confirm sorting by author, title, etc passes the same trials -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #181183|0 |1 is obsolete| | --- Comment #26 from David Nind <david@davidnind.com> --- Created attachment 181214 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=181214&action=edit Bug 23875: Explicitly srt searches by score and provide local-number as tie breaker To test: 1 - Set search engine to ES 2 - Search the catalog for "*" 3 - Note results 4 - Edit first result 5 - Repeat search, results are reordered 6 - Apply patch 7 - Search for "*" 8 - Highest biblionumbers are returned first 9 - Edit first result, repeat search 10 - Order does not change 11 - Confirm sorting by author, title, etc passes the same trials Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the| |This fixes search results release notes| |when using Elasticsearch so | |that: | |- When searching by | |relevance (the default) and | |a generic search (such as | |"*"), search results are | |returned with the highest | |record number first | |(previously, search results | |were returned with the | |lowest record number first) | |- Editing a record and | |repeating a generic search | |doesn't change the sort | |order (previously editing a | |record would change the | |search results order). | | | |Changing the sort order of | |results (such as by author | |(A-Z) or title) continue to | |work as expected. --- Comment #27 from David Nind <david@davidnind.com> --- I've attempted a release not, but not really sure I got this right! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #181214|0 |1 is obsolete| | --- Comment #28 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 181289 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=181289&action=edit Bug 23875: Explicitly srt searches by score and provide local-number as tie breaker To test: 1 - Set search engine to ES 2 - Search the catalog for "*" 3 - Note results 4 - Edit first result 5 - Repeat search, results are reordered 6 - Apply patch 7 - Search for "*" 8 - Highest biblionumbers are returned first 9 - Edit first result, repeat search 10 - Order does not change 11 - Confirm sorting by author, title, etc passes the same trials Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact| |martin.renvoize@openfifth.c | |o.uk Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 --- Comment #29 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Sounds like a good improvement, thanks! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 --- Comment #30 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Fixing authorship to avoid another mailmap: From: Nick <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to main Version(s)| |25.05.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 --- Comment #31 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Pushed for 25.05! Well done everyone, thank you! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 Jonathan Druart <jonathan.druart@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |additional_work_needed --- Comment #32 from Jonathan Druart <jonathan.druart@gmail.com> --- Koha_Main/3227 is failing 10:42:56 koha_1 | [Request] ** [http://es:9200]-[400] [query_shard_exception] No mapping found for [local-number] in order to sort on, with: {"index":"koha_kohadev_mydb","index_uuid":"Gof8vi8fTLy1hzF1wmgyDw"}, called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Search.pm line 98. With vars: {'body' => {'error' => {'type' => 'search_phase_execution_exception','failed_shards' => [{'node' => '9YF9t7UURligLeU9lgwFog','index' => 'koha_kohadev_mydb','shard' => 0,'reason' => {'index_uuid' => 'Gof8vi8fTLy1hzF1wmgyDw','reason' => 'No mapping found for [local-number] in order to sort on','type' => 'query_shard_exception','index' => 'koha_kohadev_mydb'}}],'phase' => 'query','root_cause' => [{'type' => 'query_shard_exception','index' => 'koha_kohadev_mydb','index_uuid' => 'Gof8vi8fTLy1hzF1wmgyDw','reason' => 'No mapping found for [local-number] in order to sort on'}],'reason' => 'all shards failed','grouped' => bless( do{\(my $o = 1)}, 'JSON::PP::Boolean' )},'status' => 400},'request' => {'mime_type' => 'application/json','path' => '/koha_kohadev_mydb/_search','ignore' => [],'method' => 'GET','serialize' => 'std','body' => {'size' => 20,'sort' => [{'_score' => {'order' => 'desc'}},{'local-number' => {'order' => 'desc'}}],'aggregations' => {'su-geo' => {'terms' => {'size' => '20','field' => 'su-geo__facet'}},'ln' => {'terms' => {'field' => 'ln__facet','size' => '20'}},'ccode' => {'terms' => {'field' => 'ccode__facet','size' => '20'}},'location' => {'terms' => {'size' => '20','field' => 'location__facet'}},'author' => {'terms' => {'size' => '20','field' => 'author__facet'}},'title-series' => {'terms' => {'size' => '20','field' => 'title-series__facet'}},'itype' => {'terms' => {'size' => '20','field' => 'itype__facet'}},'holdingbranch' => {'terms' => {'size' => '20','field' => 'holdingbranch__facet'}},'subject' => {'terms' => {'field' => 'subject__facet','size' => '20'}},'homebranch' => {'terms' => {'size' => '20','field' => 'homebranch__facet'}}},'from' => 0,'query' => {'query_string' => {'query' => 'easy','analyze_wildcard' => $VAR1->{'body'}{'error'}{'grouped'},'fields' => [],'fuzziness' => 'auto','lenient' => $VAR1->{'body'}{'error'}{'grouped'},'default_operator' => 'AND','type' => 'cross_fields'}}},'qs' => {'track_total_hits' => 'true'}},'status_code' => 400} 10:42:56 koha_1 | Unable to perform your search. Please try again. 10:42:56 koha_1 | # Looks like your test exited with 255 just after 4. 10:42:56 koha_1 | [08:42:01] t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 --- Comment #33 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Thanks, Joubu. Nick, can we get a quick fix? Let us know if we can help (lots of patches to push...!) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 --- Comment #34 from Jonathan Druart <jonathan.druart@gmail.com> --- Created attachment 181892 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=181892&action=edit Bug 23875: Fix tests -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 Jonathan Druart <jonathan.druart@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|additional_work_needed | --- Comment #35 from Jonathan Druart <jonathan.druart@gmail.com> --- (In reply to Jonathan Druart from comment #34)
Created attachment 181892 [details] [review] Bug 23875: Fix tests
Patch pushed. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 Andrii Nugged <nugged@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nugged@gmail.com --- Comment #36 from Andrii Nugged <nugged@gmail.com> --- (In reply to Ere Maijala from comment #2)
Should we care that the sorting by local-number.raw isn't numeric?
was that comment not included in decision? - - - On 25.05 Elastic (7) started to crash now, with [illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [local-number] in order to load field data by uninverting the inverted index. Note that this can use significant memory. so I returned to community's "number" from mappings.yaml, but my productions, which used 'stdno' for local-number, now became dysfunctional when searching for local-number, because numbers usually are "979-0-2307-9725-2" for example... Just no results. Without part of this patch (I especially removed line # push @{ $res->{sort} }, { 'local-number' => { order => 'desc' } }; ) all works again. Does this patch ignored that "local-number.raw isn't numeric" or - or do I need reconfigure somehow Elastic? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 --- Comment #37 from Andrii Nugged <nugged@gmail.com> --- I ended up with push @{ $res->{sort} }, { 'local-number.raw' => { order => 'desc' } }; hotfix instead, and it works. But want to know how that properly should be done globally. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |40658 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40658 [Bug 40658] When sorting by local-number we should use the sort field -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 --- Comment #38 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- (In reply to Andrii Nugged from comment #37)
I ended up with
push @{ $res->{sort} }, { 'local-number.raw' => { order => 'desc' } };
hotfix instead, and it works. But want to know how that properly should be done globally.
See bug 40658 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 --- Comment #39 from Andrii Nugged <nugged@gmail.com> --- (In reply to Nick Clemens (kidclamp) from comment #38)
See bug 40658
Thanks! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23875 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to main |Needs documenting --- Comment #40 from Fridolin Somers <fridolin.somers@biblibre.com> --- I prefer not impact 24.11.x LTS for stability and because of work in progress in linked Bug 40658 -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org