https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 --- Comment #1 from Lari Strand <lmstrand@gmail.com> --- Created attachment 167110 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=167110&action=edit Bug 36947 - Diacritics and lower case letters not taken into account when facets are alphabetized When using facets with Elasticsearch, the facets are alphabetized with upper and lower case letters and diacritics being separated in the alphabetizing process. This patch fixes that problem. To test: 1) You will need records with diacritics as authors/keywords like "Šostakovitš, Dmitri". 2) Reindex Elasticsearch if needed 3) Observe facets, notice author/keyword "Šostakovitš, Dmitri" ends up at the end of the facets instead adjacent to entries starting with letter S and lower case keywords being separated from upper case keywords 4) Apply patch. 5) Observe the facets' entries get alphabetized correctly. 6) Sign off. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 Lari Strand <lmstrand@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #167110|0 |1 is obsolete| | --- Comment #2 from Lari Strand <lmstrand@gmail.com> --- Created attachment 167125 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=167125&action=edit Bug 36947 - Diacritics and lower case letters not taken into account when facets are alphabetized When using facets with Elasticsearch, the facets are alphabetized with upper and lower case letters and diacritics being separated in the alphabetizing process. This patch fixes that problem. To test: 1) You will need records with diacritics as authors/keywords like "Šostakovitš, Dmitri". 2) Index Elasticsearch if needed, set syspref FacetOrder to "alphabetically" 3) Observe facets, notice author/keyword "Šostakovitš, Dmitri" ends up at the end of the facets instead adjacent to entries starting with letter S and lower case keywords being separated from upper case keywords 4) Apply patch. 5) Observe the facets' entries get alphabetized correctly. 6) Sign off. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au --- Comment #3 from David Cook <dcook@prosentient.com.au> --- Oh this sounds interesting. -- I agree about Unicode::Normalize being the way to go for the diacritics... As for the normalization form... a quick Google suggest that NFKD is most likely the correct normalization form to use, although it might only help in terms of the initial sorting based on the first letter. For instance, I think ÅB should become A + a combining ring above (030A bytes) + B. I'm actually running some test code using Unicode::Normalize, and I can't get NFKD() to work at all... going to try out some more things... -- I'm not sure what I think about the case insensitivity. I could imagine different libraries possibly having different views on that one. That said, I think a default of forcing all to uppercase or lowercase would be OK. I wonder if there's any consequences of NFKD being used after the uc() rather than the other way around, but I can't think of any off the top of my head. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 --- Comment #5 from David Cook <dcook@prosentient.com.au> --- I'm going to send a follow-up quick... and you let me know if it works for you. I should be having lunch right now, but I find this interesting. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 --- Comment #6 from David Cook <dcook@prosentient.com.au> --- Created attachment 167141 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=167141&action=edit Bug 36947: Remove diacritics from decomposed strings in ES search This patch removes the diacritics from strings decomposed using Unicode::Normalize's NFKD function. It's necessary to provide equivalent sorting of accented and unaccented characters. Test plan: 0. Apply patch 1. koha-plack --restart kohadev 2. Switch to using Elasticsearch and reindex koha-elasticsearch -b -v --rebuild kohadev 3. Setup some test records with authors with accented and unaccented names, and different cases for the lead letter e.g. Aa author, Åa author2, aa author3 4 Do a test search e.g. http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=test 5. Confirm the facet names are sorted in ascending order regardless of accent or case e.g. Aa author Åa author2 aa author3 Farley, David Humble, Jez -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #167141|0 |1 is obsolete| | --- Comment #8 from David Cook <dcook@prosentient.com.au> --- Created attachment 167143 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=167143&action=edit Bug 36947: Remove diacritics from decomposed strings in ES search This patch removes the diacritics from strings decomposed using Unicode::Normalize's NFKD function. It's necessary to provide equivalent sorting of accented and unaccented characters. Test plan: 0. Apply patch 1. koha-plack --restart kohadev 2. Switch to using Elasticsearch and reindex koha-elasticsearch -b -v --rebuild kohadev 3. Setup some test records with authors with accented and unaccented names, and different cases for the lead letter e.g. Aa author, Åa author2, aa author3 4 Do a test search e.g. http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=test 5. Confirm the facet names are sorted in ascending order regardless of accent or case e.g. Aa author Åa author2 aa author3 Farley, David Humble, Jez -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 --- Comment #10 from David Cook <dcook@prosentient.com.au> --- So I'm a bit confused... On Mattermost, @paxed is saying for Finnish "it should also sort åöä to the end of the alphabet, as per the system preference" But Lari's patch wouldn't do that, and Lari works at Koha-Suomi with @paxed? -- None of my clients have raised this as an issue yet, so I'm not super invested. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 Lari Strand <lmstrand@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #167142|0 |1 is obsolete| | --- Comment #11 from Lari Strand <lmstrand@gmail.com> --- Created attachment 167145 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=167145&action=edit Bug 36947 - Diacritics and lower case letters not taken into account when facets are alphabetized Signed-off-by: David Cook <dcook@prosentient.com.au> Signed-off-by: Lari Strand <lari.strand@koha-suomi.fi> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 --- Comment #13 from David Cook <dcook@prosentient.com.au> --- Another idea... I wonder if we could tell Perl the locale and have it do the heavy lifting. Maybe the user's locale or (as Katrin suggested) the library's locale... -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 --- Comment #15 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- We also figured out that the expectations might differ from language to language (or locale) with some of us using letters that present the same way (thinking of Ä) -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 --- Comment #17 from David Cook <dcook@prosentient.com.au> --- I'm going to try this out in a minute... -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |In Discussion -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 --- Comment #20 from David Cook <dcook@prosentient.com.au> --- And sure enough... When I just add "use locale" to sub "_convert_facets" in Koha/SearchEngine/Elasticsearch/Search.pm and make sure the environmental variable LANG is set to the locale I want (ie Finnish)... then my facets are sorted as you'd hope in Finnish. Actually... in terms of environmental variables, it's looking like it was LC_ALL=fi_FI.UTF-8 that did the trick. There's still some experimenting to do here... But the order looks right for Finnish: Aa author aa author3 étienne Farley, David Humble, Jez Martin, Robert C. Åa author2 -- So... I think that's actually the way to go. I think that's going to provide a much better experience to users. For now, we work with the server's locale, but we could potentially use the user's locale. I'm not an expert on that one, but I notice my browser sends "Accept-Language" and includes en-GB and en-US. So there could be more interesting things done there. Anyway, alternate patch incoming... -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 --- Comment #22 from David Cook <dcook@prosentient.com.au> --- Created attachment 167188 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=167188&action=edit Bug 36947: [Alternate] Do a locale-based sort for ES facet names This change causes the locale system to be used when sorting for ES facet names. Test plan: 0. Apply the patch 1. vi "/etc/default/koha-common" 2. Add the following to the bottom of the file: export LC_ALL=fi_FI.UTF-8 3. koha-plack --restart koha-common 4. Setup some test records with authors with accented and unaccented names, and different cases for the lead letter e.g. Aa author, Åa author2, aa author3, étienne 5. Switch to using Elasticsearch and reindex koha-elasticsearch -b -v --rebuild kohadev 6. Do a test search e.g. http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=test 7. Confirm the facet names are sorted in ascending order following Finnish collation rules e.g. Aa author aa author3 étienne Farley, David Humble, Jez Martin, Robert C. Åa author NOTE: Any collation and language can be used. Finnish is just an example of a Latin-based script which has a different alphabetical ordering than just A-Z -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 --- Comment #24 from Lari Strand <lmstrand@gmail.com> --- We're happy with this solution. I can sign off. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m --- Comment #26 from Fridolin Somers <fridolin.somers@biblibre.com> --- +use Unicode::Normalize; Dont we need to choose the method ? +use Unicode::Normalize qw( NFKD ); -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 --- Comment #28 from David Cook <dcook@prosentient.com.au> --- (In reply to Fridolin Somers from comment #26)

+use Unicode::Normalize; Dont we need to choose the method ? +use Unicode::Normalize qw( NFKD );

Unicode::Normalize exports NFC, NFD, NFKC, and NFKD by default. Probably would be better to just export NFKD, but shouldn't matter too much. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 --- Comment #30 from David Cook <dcook@prosentient.com.au> --- Of course, there's always this 3rd option of using "Unicode::Collate::Locale", which I've called "Alternate 2". I think that I like this option best, as it's the most configurable. You can just feed whatever locale you want into the object and then have it do the sort for you. In my patch, I have it fetch LC_COLLATE, but in the future we could base it off of C4::Languages::getlanguage($cgi). There's options, and I like having options. I suspect in the future that there will be other places that we want to sort using locale as well, and I think Unicode::Collate::Locale will be the most portable for those situations. -- Lari, I suspect that this will work for you better than the one using "use locale". Please give it a try and let me know what you think. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 --- Comment #32 from Lari Strand <lmstrand@gmail.com> --- This seems to achieve the same result as use locale (when LC_COLLATE is set to "fi_FI"): if( C4::Context->preference('FacetOrder') eq 'Alphabetical' ){ my $coll = Unicode::Collate::Locale->new( locale => "fi_FI" ); @{ $facet->{facets} } = sort { $coll->cmp ($a->{facet_label_value}, $b->{facet_label_value}) } @{ $facet->{facets} }; } push @facets, $facet if exists $facet->{facets}; Should the collator be configurable in Koha or were you thinking of using the selected UI language for sorting somehow here? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 --- Comment #34 from Lari Strand <lmstrand@gmail.com> --- Sorry, missed the patch :D -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 --- Comment #36 from Lari Strand <lmstrand@gmail.com> --- Found couple of bugs linking to this one. Collections get alphabetized by the auth value code rather than the description. Will make another bugzilla ticket for this unless I can find one. I already have a fix for this. Also languages get alpabetized based on their english descriptions instead of the selected intranet language. This might be more difficult to fix... -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 --- Comment #38 from David Cook <dcook@prosentient.com.au> --- (In reply to Lari Strand from comment #36)

Also languages get alpabetized based on their english descriptions instead of the selected intranet language. This might be more difficult to fix...

I think that relates to this bug 36947, or at least the solution is related. I think Unicode::Collate::Locale would work well for that. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 --- Comment #40 from Lari Strand <lmstrand@gmail.com> --- I like alternative 2, Do a locale-based sort for ES facet names" best as well. We actually implemented this in our production environment with my $coll = Unicode::Collate::Locale->new( locale => "fi_FI" ); and using it in the sorting phase, just to avoid making adjustments to our 9 production environments which run different consortiums' Koha installations in parallel, I was lazy :D. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 --- Comment #42 from Lari Strand <lmstrand@gmail.com> --- I guess there could be cases where you want to override your env LC_COLLATE locale with one set in koha-conf.xml or Apache. I'm not familiar on how Koha uses localizations in other ways besides selecting intranet language based on your system locale (LC_LANG?). I've seen some code regarding monetary units like [% fines | $Price with_symbol => 1 %], don't know where this can be configured though. I don't think env locale has anything to do with that, otherwise we would be playing with USD instead of €(eur) atm (LC_* = en_US). "Or maybe it should be an environmental variable that we can set in the Apache configuration, so that different OPACs can provide different values." I just want to see all Koha side localization configuration/overrides in the same place if possible. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #167145|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #167188|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 Lari Strand <lmstrand@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #167198|0 |1 is obsolete| | --- Comment #45 from Lari Strand <lmstrand@gmail.com> --- Created attachment 167880 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=167880&action=edit Bug 36947: [Alternate 2] Do a locale-based sort for ES facet names This change uses a configurable locale-based collator to sort the ES facet names. Test plan: 0. Apply the patch 1. vi "/etc/default/koha-common" 2. Add the following to the bottom of the file: export LC_ALL=fi_FI.UTF-8 3. koha-plack --restart koha-common 4. Setup some test records with authors with accented and unaccented names, and different cases for the lead letter e.g. Aa author, Åa author2, aa author3, étienne 5. Switch to using Elasticsearch and reindex koha-elasticsearch -b -v --rebuild kohadev 6. Do a test search e.g. http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=test 7. Confirm the facet names are sorted in ascending order following Finnish collation rules e.g. Aa author aa author3 étienne Farley, David Humble, Jez Martin, Robert C. Åa author NOTE: Any collation and language can be used. Finnish is just an example of a Latin-based script which has a different alphabetical ordering than just A-Z Signed-off-by: Lari Strand <lari.strand@koha-suomi.fi> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 --- Comment #46 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 168488 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=168488&action=edit Bug 36947: Do a locale-based sort for ES facet names This change uses a configurable locale-based collator to sort the ES facet names. Test plan: 0. Apply the patch 1. vi "/etc/default/koha-common" 2. Add the following to the bottom of the file: export LC_ALL=fi_FI.UTF-8 3. koha-plack --restart koha-common 4. Setup some test records with authors with accented and unaccented names, and different cases for the lead letter e.g. Aa author, Åa author2, aa author3, étienne 5. Switch to using Elasticsearch and reindex koha-elasticsearch -b -v --rebuild kohadev 6. Do a test search e.g. http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=test 7. Confirm the facet names are sorted in ascending order following Finnish collation rules e.g. Aa author aa author3 étienne Farley, David Humble, Jez Martin, Robert C. Åa author NOTE: Any collation and language can be used. Finnish is just an example of a Latin-based script which has a different alphabetical ordering than just A-Z Signed-off-by: Lari Strand <lari.strand@koha-suomi.fi> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #167880|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Failed QA --- Comment #47 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- This is a great first pass... I'd love to see an option added to koha-conf.xml to allow setting at the instance level. More importantly though.. if we can add a unit test here I think we could PQA. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 --- Comment #49 from David Cook <dcook@prosentient.com.au> --- And as for the koha-conf.xml option... yeah cool. I think that makes sense for the first implementation. I'm so excited to explore more options in the future as well. I love languages and UTF-8 :D. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 --- Comment #50 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Would love to see this over the finish line ;) -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 --- Comment #52 from David Cook <dcook@prosentient.com.au> --- (In reply to Martin Renvoize from comment #50)

Would love to see this over the finish line ;)

I'm hoping to look at this in August... -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 --- Comment #54 from David Cook <dcook@prosentient.com.au> --- (In reply to David Cook from comment #53)

But I'll have a go again with Finnish and have another look when it's not so late...

I was going to leave this for another day but I'm just too stubborn. 1. vi /etc/locale.gen 2. Uncomment the locale you want to generate (e.g. fi_FI.UTF-8 UTF-8) 3. locale-gen Now when I give Unicode::Collate::Locale the "fi_FI" locale, it definitely sets it. That suggests to me that both English and French use the same collator. That's going to make unit testing this in any meaningful way pretty hard... but maybe that just means I need to raise a ticket with koha-testing-docker folk to add fi_FI as a locale to ktd so that we can do this kind of testing... blah... -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Diacritics and upper case |Sort Elasticsearch facets |letters should not affect |according to system locale |how facets get alphabetized |instead of using Perl's | |stringwise/bytewise sort -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 Janusz Kaczmarek <januszop@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |januszop@gmail.com --- Comment #58 from Janusz Kaczmarek <januszop@gmail.com> --- *** Bug 38896 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 --- Comment #60 from David Cook <dcook@prosentient.com.au> --- (In reply to Janusz Kaczmarek from comment #59)

It seems that the issue affects in the same way also Zebra facets--also in C4::Search::getRecords (at the end) facets' content is sorted by a simple sort.

Shouldn't we fix it in one step?

Sure. Originally, I was just updating Lari's patch to use "Unicode::Collate::Locale" so I didn't even look at the Zebra side of things. Happy for you to take this one over if you like. Most/all of my libraries primarily use English/French, and most of my libraries use Elasticsearch, so this one isn't a high priority for me at the moment. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 --- Comment #62 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 183552 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=183552&action=edit Bug 36947: Add system preference for configurable locale-based facet sorting This follow-up adds two major enhancements to the locale-based facet sorting implementation: 1. **System Preference Integration** - Adds FacetSortingLocale system preference for configurable locale selection - Dynamically detects available system locales using Koha::I18N - Provides user-friendly dropdown with locale descriptions - Supports both Elasticsearch and Zebra search engines 2. **Zebra Search Engine Support** - Extends locale-based sorting to Zebra (in addition to existing Elasticsearch) - Implements _sort_facets_zebra() function in C4::Search - Maintains consistent Unicode-aware sorting behavior across search engines Key Features: - Dynamic locale detection via Koha::I18N::available_locales() - Proper module architecture with comprehensive unit tests - Clean fallback chain: preference → system LC_COLLATE → default - Admin UI integration in System Preferences → I18N/L10N - Universal facet sorting for international library systems Test plan: 1. Apply patch and restart services 2. Install/run database update 3. Navigate to Administration → System Preferences → I18N/L10N 4. Verify "Sort facet names using" dropdown shows available system locales 5. Test facet sorting with different locales and both search engines 6. Run: prove t/Koha/I18N.t t/Koha/SearchEngine/Elasticsearch/Search.t t/db_dependent/Search_FacetSorting.t -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 --- Comment #64 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- OK.. I've done some pretty extensive follow on work here to try and get us over the line. Please give it a good testing. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA --- Comment #66 from David Cook <dcook@prosentient.com.au> --- Regarding "Dynamic locale detection via Koha::I18N::available_locales()", I think that this isn't working at the moment, as FacetSortingLocale is showing an empty list. While I haven't generated any locales yet, "locale -a" is showing C, C.utf8, en_US.utf8, fr_FR.utf8, and POSIX, plus in Koha::I18N::available_locales() it seems like you have a the default option which should always be appearing, so I think something isn't working right. I'm going to marked as "Failed QA", and I'm going to spend a few minutes seeing if I can figure out what's going on there... -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 --- Comment #68 from David Cook <dcook@prosentient.com.au> --- Created attachment 183563 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=183563&action=edit Bug 36947: (QA follow-up) Fix hash key name for locale-list syspref type -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 --- Comment #70 from David Cook <dcook@prosentient.com.au> --- My test plan (before patch): - Go to http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=29 - Add 700 entries for the following list of authors: - Aa author - Åa author2 - aa author - étienne - Go to http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=test - Note following sort order of Authors facet: Aa author aa author Farley, David Humble, Jez Martin, Robert C. Åa author2 étienne My test plan (with patch): - Apply patch - koha-upgrade-schema kohadev - service koha-common restart - Go to http://localhost:8081/cgi-bin/koha/admin/preferences.pl?op=search&searchfield=SearchEngine - Confirm that the SearchEngine syspref is set to "Zebra" - Go to http://localhost:8081/cgi-bin/koha/admin/preferences.pl?op=search&searchfield=FacetSortingLocale - Note that entries appear in the dropdown list for "FacetSortingLocale" including: - Default Unicode collation - English (United States) - en_US.utf8 - French (France) - fr_FR.utf8 - Go to http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=test - Note following sort order of Authors facet: aa author Aa author Åa author2 étienne Farley, David Humble, Jez Martin, Robert C. - Go to http://localhost:8081/cgi-bin/koha/admin/preferences.pl?op=search&searchfield=SearchEngine - Change SearchEngine syspref to "Elasticsearch" - service koha-common restart - koha-elasticsearch --rebuild kohadev -v - Go to http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=test - Note following sort order of Authors facet: aa author Aa author Åa author2 étienne Farley, David Humble, Jez Martin, Robert C. - Change the "FacetSortingLocale" from "Default Unicode collation" to "English (United States) - en_US.utf8" and retry search - Note the sort order is same as "Default Unicode collation" - Change the "FacetSortingLocale" from "Default Unicode collation" to "French (France) - fr_FR.utf8" and retry search - Note the sort order is same as "Default Unicode collation" -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA --- Comment #72 from David Cook <dcook@prosentient.com.au> --- This is really awesome. It works very well! Thanks so much for saving it, Martin. Unfortunately, I found 1 more bug. While my follow-up fixed FacetSortingLocale so that the select list entries now appear... your saved selection doesn't appear in the UI. Your selection does get saved to the database, but the select list won't show it after a page reload. I'm going to take a look at that quick to see if I can fix that, but until then "Failed QA" again... -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #183552|0 |1 is obsolete| | --- Comment #74 from David Cook <dcook@prosentient.com.au> --- Created attachment 183565 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=183565&action=edit Bug 36947: Add system preference for configurable locale-based facet sorting This follow-up adds two major enhancements to the locale-based facet sorting implementation: 1. **System Preference Integration** - Adds FacetSortingLocale system preference for configurable locale selection - Dynamically detects available system locales using Koha::I18N - Provides user-friendly dropdown with locale descriptions - Supports both Elasticsearch and Zebra search engines 2. **Zebra Search Engine Support** - Extends locale-based sorting to Zebra (in addition to existing Elasticsearch) - Implements _sort_facets_zebra() function in C4::Search - Maintains consistent Unicode-aware sorting behavior across search engines Key Features: - Dynamic locale detection via Koha::I18N::available_locales() - Proper module architecture with comprehensive unit tests - Clean fallback chain: preference → system LC_COLLATE → default - Admin UI integration in System Preferences → I18N/L10N - Universal facet sorting for international library systems Test plan: 1. Apply patch and restart services 2. Install/run database update 3. Navigate to Administration → System Preferences → I18N/L10N 4. Verify "Sort facet names using" dropdown shows available system locales 5. Test facet sorting with different locales and both search engines 6. Run: prove t/Koha/I18N.t t/Koha/SearchEngine/Elasticsearch/Search.t t/db_dependent/Search_FacetSorting.t Signed-off-by: David Cook <dcook@prosentient.com.au> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #183563|0 |1 is obsolete| | --- Comment #76 from David Cook <dcook@prosentient.com.au> --- Created attachment 183567 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=183567&action=edit Bug 36947: (QA follow-up) Fix hash key name for locale-list syspref type Signed-off-by: David Cook <dcook@prosentient.com.au> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Sort Elasticsearch facets |Sort Elasticsearch/Zebra |according to system locale |facets according to |instead of using Perl's |configurable locale instead |stringwise/bytewise sort |of using Perl's | |stringwise/bytewise sort -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 Jonathan Druart <jonathan.druart@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@gmail.com Status|Signed Off |In Discussion --- Comment #81 from Jonathan Druart <jonathan.druart@gmail.com> --- Are you sure using Sort::Naturally is not enough? At least the tests pass. If it's not enough please provide a test that makes this last patch fail. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |tomascohen@gmail.com --- Comment #83 from David Cook <dcook@prosentient.com.au> --- (In reply to Jonathan Druart from comment #81)

Are you sure using Sort::Naturally is not enough? At least the tests pass. If it's not enough please provide a test that makes this last patch fail.

Yes, I am reasonably sure that Sort::Naturally is not enough. If you read through the comments, you'll see that I've previously indicated that the tests are flawed in the current koha-testing-docker because of the available locales. This could be fixed, but would take some coordination with whoever manages the ktd images. I'm guessing Tomas? I'll explain in a bit more detail below. -- Unicode::Collate::Locale is core Perl. Sort::Naturally is a CPAN module which isn't currently available in Koha. Sort::Naturally appears to just use "use locale" under the hood. If you look back on the comments, you can see that has already been explored. Maybe I wasn't clear in past comments abouts its limitations, but from memory it's limited to the system locale or the locale as specified in the runtime environment of that process at startup (I can't recall if it's compile time or run time though). Unicode::Collate::Locale allows us to very easily work with locales supplied at runtime, which creates many wonderful possibilities for adapting to the language needs of users. Due to limited resources, Martin has specified the locale via a system preference, but there's really a lot more potential here, if you're interested in this topic. Finally, the unit tests are constrained by the locales available on koha-testing-docker. If you run `locale -a`, you'll see the following: C C.utf8 en_US.utf8 fr_FR.utf8 POSIX All of these locales use the same underlying Unicode collator, so they're all always going to have the same result. In order for Martin or myself to "provide a test that makes this last patch fail", we'd have to change the koha-testing-docker to include a locale like fi_FI which has a different Unicode collator. Now I think it would be a great idea to add fi_FI to koha-testing-docker for this purpose alone. I think we haven't done that because of the obstacle that presents, but yeah... that's an option. Probably a good one in fact. I'm adding Tomas as CC. I'll send him a message on Mattermost too. If we can get a new KTD image with fi_FI as an available locale, then we can write a test to prove how good Unicode::Collate::Locale is and why we need it. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 --- Comment #85 from David Cook <dcook@prosentient.com.au> --- Another reason why you'd use Unicode::Collate::Locale and not "use locale" is so that 1 Koha server can provide Koha instances with different locales. Technically, you could use both "use locale" and "setlocale()" together I suppose, but again Unicode::Collate::Locale is a nice core library designed for this exact purpose that we're using it for. (In reply to David Cook from comment #57)

I just finished writing and polishing this...

And I realize the LC_COLLATE thing is controversial, since it's a system-level thing and not an instance-level thing.

A system preference or a koha-conf.xml config is probably the way to go. We'd look it up in Koha/SearchEngine/Elasticsearch/Search.pm and pass it into the _sort_facets() function using the "locale" parameter.

--

I was going to say that since it's a system level thing and something that shouldn't be changed by librarians, I thought koha-conf.xml would be better.

But... if we used the system preference, then people could override it using VirtualHost directives in Apache, so that different front-ends use different locales.

I'm really not too fussed... the main reason I did this patch was out of the goodness of my heart (or perhaps intellectual vanity or perhaps both), so I'm happy to be led by others here on the configuration side of things.

-- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #183727|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 --- Comment #88 from David Cook <dcook@prosentient.com.au> --- Created attachment 183818 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=183818&action=edit Bug 36947: (QA follow-up) Add skippable unit test for fi_FI locale This patch adds a skippable unit test for the fi_FI locale to prove that the patches work well for other locales that don't use the default Unicode collation. It will run on koha-testing-docker, but will be skipped on systems lacking it. This patch also fleshes out the existing unit tests a bit more to prove that the patches also improves the sorting of names with diacritics generally. That is, the default Unicode sort is better than the existing stringwise/bytewise sort even for English and French locales. Test plan: 0. prove -v t/Koha/SearchEngine/Elasticsearch/Search.t \ t/db_dependent/Search_FacetSorting.t -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #183817|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #183565|0 |1 is obsolete| | --- Comment #90 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 183882 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=183882&action=edit Bug 36947: Add system preference for configurable locale-based facet sorting This follow-up adds two major enhancements to the locale-based facet sorting implementation: 1. **System Preference Integration** - Adds FacetSortingLocale system preference for configurable locale selection - Dynamically detects available system locales using Koha::I18N - Provides user-friendly dropdown with locale descriptions - Supports both Elasticsearch and Zebra search engines 2. **Zebra Search Engine Support** - Extends locale-based sorting to Zebra (in addition to existing Elasticsearch) - Implements _sort_facets_zebra() function in C4::Search - Maintains consistent Unicode-aware sorting behavior across search engines Key Features: - Dynamic locale detection via Koha::I18N::available_locales() - Proper module architecture with comprehensive unit tests - Clean fallback chain: preference → system LC_COLLATE → default - Admin UI integration in System Preferences → I18N/L10N - Universal facet sorting for international library systems Test plan: 1. Apply patch and restart services 2. Install/run database update 3. Navigate to Administration → System Preferences → I18N/L10N 4. Verify "Sort facet names using" dropdown shows available system locales 5. Test facet sorting with different locales and both search engines 6. Run: prove t/Koha/I18N.t t/Koha/SearchEngine/Elasticsearch/Search.t t/db_dependent/Search_FacetSorting.t Signed-off-by: David Cook <dcook@prosentient.com.au> Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #183567|0 |1 is obsolete| | --- Comment #92 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 183884 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=183884&action=edit Bug 36947: (QA follow-up) Fix hash key name for locale-list syspref type Signed-off-by: David Cook <dcook@prosentient.com.au> Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #183818|0 |1 is obsolete| | --- Comment #94 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 183886 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=183886&action=edit Bug 36947: (QA follow-up) Add skippable unit test for fi_FI locale This patch adds a skippable unit test for the fi_FI locale to prove that the patches work well for other locales that don't use the default Unicode collation. It will run on koha-testing-docker, but will be skipped on systems lacking it. This patch also fleshes out the existing unit tests a bit more to prove that the patches also improves the sorting of names with diacritics generally. That is, the default Unicode sort is better than the existing stringwise/bytewise sort even for English and French locales. Test plan: 0. prove -v t/Koha/SearchEngine/Elasticsearch/Search.t \ t/db_dependent/Search_FacetSorting.t Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 --- Comment #96 from Lucas Gass (lukeg) <lucas@bywatersolutions.com> --- Nice work everyone! Pushed to main for 25.11 -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 --- Comment #97 from Paul Derscheid <paul.derscheid@lmscloud.de> --- Nice work everyone! Pushed to 25.05.x -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36947 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |41287 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41287 [Bug 41287] Using locale sorting may have a negative impact on search speeds -- You are receiving this mail because: You are watching all bug changes.