[Bug 39001] New: Language list is displayed in wrong order (getLanguages doesn't sort it, the order is UB)
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39001 Bug ID: 39001 Summary: Language list is displayed in wrong order (getLanguages doesn't sort it, the order is UB) Change sponsored?: --- Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: trivial Priority: P5 - low Component: OPAC Assignee: oleonard@myacpl.org Reporter: schodkowy.omegi-0r@icloud.com QA Contact: testopia@bugs.koha-community.org Caught by unit tests added in Bug 38646, but it's not a regression of that bug. It seems that the list of languages from getLanguages is not ordered in any way, instead it relies on the implementation detail of the database server in terms of the order it returns things in. Indeed even before that bug was merged, in Koha 24.11 and 24.05, one can observe in OPAC -> Advanced search -> Language, that the order of languages is indeed a bit wrong (for example: amh, grc, ara -- Old Greek (grk) has nothing starting with "a", so it has no business being this high in the list). The solution in principle is simple, just order the returned language list (TBD what to order it by: by language code, by English name, by effective final display name????). It's also TBD whether getLanguages should sort it itself, or whether places that display languages, such as `/cgi-bin/koha/opac-search.pl`, should actually be doing the ordering in preferred fashion. In any case, the tests at t/db_dependent/Languages.t fail due to wrong order after some MariaDB update, so current order is undefined behavior. Also if we select to change the behavior of getLanguages, then it'll be trickier to backport it, since dependent Bug 38646 is enhancement due to slight change of behavior. But the backporter could easily adjust the SQL query there to make the lines "THEN CONCAT(" be selected with removing the conditional, so that the language list is always appended in parentheses (old behavior). -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39001 Michał <schodkowy.omegi-0r@icloud.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=38646 Patch complexity|--- |Trivial patch CC| |jonathan.druart@gmail.com, | |martin.renvoize@ptfs-europe | |.com --- Comment #1 from Michał <schodkowy.omegi-0r@icloud.com> --- Also wasn't sure whether to mark it as OPAC or I18N category. Technically the user-facing display bug is in OPAC, even though also unit test regressed after MariaDB update due to the UB. Welp. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39001 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39001 --- Comment #2 from David Cook <dcook@prosentient.com.au> --- I was going to say that sorting the @languages_list at the end of getLanguages() would be the easiest solution, but that wouldn't be translation friendly... ...but it looks like we just use the description straight from the database, so that's fine. I don't recall if sort() can handle UTF-8 or not. I think it should. But if not it might be necessary to use something like Unicode::Collate's sort() -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39001 --- Comment #3 from David Cook <dcook@prosentient.com.au> --- Or do we sort by code? Using Unicode::Collate might be useful with the description because that would use whatever the default is used in the Unicode Collation Algorithms... I wonder if Greek speakers would expect to find Greek at the end of the list or in the Es... This could be another case where Unicode::Collate::Locale could be useful although it runs into issues of installed locales... see bug 36947 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39001 Michał <schodkowy.omegi-0r@icloud.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |In Discussion --- Comment #4 from Michał <schodkowy.omegi-0r@icloud.com> --- Now after the follow-up to Bug 38646, the order of languages returned from C4::Languages::getLanguages is sorted by language code ascending. I think that's a good call, since it makes the output of the function stable, regardless of the session context (and the user language selected therein). So as far as undefined behavior went, that bug is solved. However, I keep it open and change status to "In Discussion", unless someone decides to close, because there might still be some merit to considering language order in places where users see it. I think for places where actual users sees a list of languages, they might really expect to see them ordered alphabetically by how they're displayed, but for that we might want to handle this directly at the places that display such lists in/for UI and not at getLanguages func level. That might be the most stable approach. Aforementioned places is mostly the advanced search form page, I think. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39001 Michał <schodkowy.omegi-0r@icloud.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Language list is displayed |Language list is displayed |in wrong order |in wrong order |(getLanguages doesn't sort |(getLanguages sorts it by |it, the order is UB) |language code; places that | |display language list to | |the end-user in UI should | |re-sort that by localized | |display name order) --- Comment #5 from Michał <schodkowy.omegi-0r@icloud.com> --- I kept the bug name -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39001 Owen Leonard <oleonard@myacpl.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |f.demians@tamil.fr Assignee|oleonard@myacpl.org |koha-bugs@lists.koha-commun | |ity.org Component|OPAC |I18N/L10N -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39001 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m Depends on| |38646 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38646 [Bug 38646] C4::Languages::getLanguages is very unreasonably slow (100+ ms) -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39001 --- Comment #6 from Fridolin Somers <fridolin.somers@biblibre.com> --- Whooo this is bad for users. There are so much languages. For example spa/Spanish is "Espagnol" in french it ends up in end of list. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39001 --- Comment #7 from Fridolin Somers <fridolin.somers@biblibre.com> --- Ahhh but I've tested this was already in Koha 23.11 with some misplaced languages. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39001 --- Comment #8 from David Cook <dcook@prosentient.com.au> --- *** Bug 28245 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39001 --- Comment #9 from David Cook <dcook@prosentient.com.au> --- (In reply to Fridolin Somers from comment #7)
Ahhh but I've tested this was already in Koha 23.11 with some misplaced languages.
Yeah I think I originally reported on bug 28245 back in 2021. I think it's been a problem for a long time. Or at least the languages haven't been sorted by display name -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39001 --- Comment #10 from David Cook <dcook@prosentient.com.au> ---
From memory, I think sorting the languages has some challenges when it comes to CJK languages?
I'm curious how other scripts like Arabic and Devanagari would sort as well. I think CJK sort by byte order which isn't great. Might be the same for other non-roman alphabet languages. But... I suppose there's nothing to do but try... -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39001 --- Comment #11 from Fridolin Somers <fridolin.somers@biblibre.com> --- Created attachment 191748 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=191748&action=edit Bug 39001: Sort language search limits by description Patch from commit d09c0b6 -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39001 --- Comment #12 from Fridolin Somers <fridolin.somers@biblibre.com> --- Here is a proposed simple fix using sort by description. Looks way better. I'll add a test plan if validated ;) -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39001 --- Comment #13 from Fridolin Somers <fridolin.somers@biblibre.com> --- This a bit like for item types : https://git.koha-community.org/Koha-community/Koha/src/commit/f21255ca54ac4c... -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39001 --- Comment #14 from Michał <schodkowy.omegi-0r@icloud.com> --- Glancing at this again, I think it could make sense to instead add a third parameter to getLanguages in C4/Languages.pm to deduplicate the sorting code and ensure the translated order is consistent in different places too? Maybe under `my $isFiltered = shift;` add something like `my $isSortedByDescription = shift;`? And then sort it in an "if" right before returning (keep the default ORDER BY in the SQL query above it). And it'd be opt-in, so it'd still default to language codes sorting as expected without explicitly wanting it sorted like for displaying in HTML template. And then as David suggested, I think the actual sorting should then use Unicode::Collate's sorting, if my memory serves me right I don't think any other ordering would be proper or sufficient. With its default config, it should be a simple one-liner just as well. And since the ES facets seem to use Unicode::Collate now too, I don't see a reason not to use it here as well.
This could be another case where Unicode::Collate::Locale could be useful although it runs into issues of installed locales... see bug 36947
I think we can ignore that, there are many places in Koha now where localization will break without these locales installed, and it's now mentioned in the wiki install guide (not sure if prominently enough, but yeah...). -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39001 --- Comment #15 from David Cook <dcook@prosentient.com.au> --- (In reply to Michał from comment #14)
And then as David suggested, I think the actual sorting should then use Unicode::Collate's sorting, if my memory serves me right I don't think any other ordering would be proper or sufficient. With its default config, it should be a simple one-liner just as well. And since the ES facets seem to use Unicode::Collate now too, I don't see a reason not to use it here as well.
I think we reverted it out of the ES facets as it was causing performance problems when large numbers of facets were used :( -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39001 --- Comment #16 from Fridolin Somers <fridolin.somers@biblibre.com> ---
I think it could make sense to instead add a third parameter to getLanguages in C4/Languages.pm +1
-- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39001 David Liddle <david_liddle@sil.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david_liddle@sil.org --- Comment #17 from David Liddle <david_liddle@sil.org> --- Hello! Is this issue what affects the different ordering of languages that I see in the Edit mode of modules such as Pages and HTML customizations? (Koha 24.11) That sorting is neither by what is specified in StaffInterfaceLanguages nor by language code (my personal preference). The inconsistency is disruptive when one is doing a lot of localization. If it's another bug altogether, please point me in the right direction. Thanks! -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39001 Bug 39001 depends on bug 38646, which changed state. Bug 38646 Summary: C4::Languages::getLanguages is very unreasonably slow (100+ ms) https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38646 What |Removed |Added ---------------------------------------------------------------------------- Status|Needs documenting |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39001 Jonathan Druart <jonathan.druart@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=42727 --- Comment #18 from Jonathan Druart <jonathan.druart@gmail.com> --- (In reply to David Liddle from comment #17)
Hello! Is this issue what affects the different ordering of languages that I see in the Edit mode of modules such as Pages and HTML customizations? (Koha 24.11)
That sorting is neither by what is specified in StaffInterfaceLanguages nor by language code (my personal preference). The inconsistency is disruptive when one is doing a lot of localization.
If it's another bug altogether, please point me in the right direction. Thanks!
Will be fixed by bug 42727. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org