[Bug 10939] New: ICU does not transliterate polish special characters
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10939 Bug ID: 10939 Summary: ICU does not transliterate polish special characters Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: minor Priority: P5 - low Component: Searching Assignee: gmcharlt@gmail.com Reporter: fridolyn.somers@biblibre.com QA Contact: testopia@bugs.koha-community.org When indexing records in Zebra using ICU (in etc/default.idx), non latin characters are transliterate to latin one. It works well with arabic for example but not for polish special characters. See http://en.wikipedia.org/wiki/Polish_alphabet#Computer_encoding -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10939 Fridolyn SOMERS <fridolyn.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |http://bugs.koha-community. | |org/bugzilla3/show_bug.cgi? | |id=2629 -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10939 --- Comment #1 from Galen Charlton <gmcharlt@gmail.com> --- Can you expand on this bug report? "Not working well" is rather vague. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10939 --- Comment #2 from Jared Camins-Esakov <jcamins@cpbibliography.com> --- (In reply to Fridolyn SOMERS from comment #0)
When indexing records in Zebra using ICU (in etc/default.idx), non latin characters are transliterate to latin one.
Are you sure about this? It sounds to me like you are describing the behavior of Zebra with charmap, not Zebra with ICU. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10939 Colin Campbell <colin.campbell@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |colin.campbell@ptfs-europe. | |com --- Comment #3 from Colin Campbell <colin.campbell@ptfs-europe.com> --- The delivered words-icu.xml does a decompose (NFD) then removes Non-spacing mark then a recompose (NFC). That may strip diacritics from characters which should have a separate collation value. Probably needs some experimentation with the transform rules to confirm -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10939 mathieu saby <mathieu.saby@univ-rennes2.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mathieu.saby@univ-rennes2.f | |r --- Comment #4 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- Hi Some years ago, I find out the polish "l stroke" was not considered as a variant of "l". Do you think of that (Fridolyn, it is in french in my blog http://www.vingtseptpointsept.fr/2011/09/22/une-fonction-javascript-pour-eli... "Je pensais utiliser ce fichier, mais il n’est malheureusement pas assez précis. Certains caractères que l’on peut légitimement considérer comme composés ne sont pas signalés comme tels. C’est le cas notamment du l barré polonais") Maybe I was wrong, I did not work on that since 2011... Mathieu -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10939 --- Comment #5 from Fridolyn SOMERS <fridolyn.somers@biblibre.com> --- (In reply to Galen Charlton from comment #1)
Can you expand on this bug report? "Not working well" is rather vague. I mean I does not work at all, polish "l stroke" is not considered as the same character has "l".
(In reply to Jared Camins-Esakov from comment #2)
(In reply to Fridolyn SOMERS from comment #0)
When indexing records in Zebra using ICU (in etc/default.idx), non latin characters are transliterate to latin one.
Are you sure about this? It sounds to me like you are describing the behavior of Zebra with charmap, not Zebra with ICU. I do not know the real behavior of Zebra+ICU, but I mean that non latin characters can be searched using latin characters and the opposite. That is the aim of ICU.
(In reply to mathieu saby from comment #4)
Hi Some years ago, I find out the polish "l stroke" was not considered as a variant of "l". Even if true, It would be easier use the "l" to search into polish records no ? "l stroke" does not exist in a QWERTY nor AZERTY keyboard.
-- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10939 --- Comment #6 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- I don't speak polish, but I agree with you Fridolyn. Maybe for Poles it is a very different letter than "l", but if we have a polish book with this stroke l in our collection, we need a way to search it with a simple "l" ;-) Mathieu -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10939 Fridolyn SOMERS <fridolyn.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff Assignee|gmcharlt@gmail.com |fridolyn.somers@biblibre.co | |m --- Comment #7 from Fridolyn SOMERS <fridolyn.somers@biblibre.com> --- Created attachment 22599 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=22599&action=edit Bug 10939 - ICU does not transliterate polish special characters Polish characters added to ICU config -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10939 --- Comment #8 from Colin Campbell <colin.campbell@ptfs-europe.com> --- Some of the added transliteration rules are default ICU behaviour, i.e. stripping accents. I'm a bit wary of this shipping as default as a polish language library would, I assume, have to undo this. And why Polish especially? What about Norwegian with its extra characters? I wonder if for release what is needed is rather some documentation on 'how to alter normalization of non-ascii characters'. It might call for a bit of research -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10939 --- Comment #9 from Fridolyn SOMERS <fridolyn.somers@biblibre.com> --- (In reply to Colin Campbell from comment #8)
I wonder if for release what is needed is rather some documentation on 'how to alter normalization of non-ascii characters'. It might call for a bit of research Ok for documentation, there is indeed such a page on wiki for arabic.
-- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10939 Fridolyn SOMERS <fridolyn.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #22599|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10939 Fridolyn SOMERS <fridolyn.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |In Discussion -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10939 --- Comment #10 from Fridolyn SOMERS <fridolyn.somers@biblibre.com> --- I set to discussion while i create the wiki page -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10939 Fridolyn SOMERS <fridolyn.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Discussion |RESOLVED Resolution|--- |INVALID --- Comment #11 from Fridolyn SOMERS <fridolyn.somers@biblibre.com> --- Done : http://wiki.koha-community.org/wiki/Correcting_Search_of_Polish_records -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10939 Jacek Ablewicz <abl@biblos.pk.edu.pl> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |abl@biblos.pk.edu.pl --- Comment #12 from Jacek Ablewicz <abl@biblos.pk.edu.pl> --- I think you only need those two lines in etc/words-icu.xml: <transliterate rule="{ ł > l "/> <transliterate rule="{ Ł > l "/> as all other Polish diacritics should behave correctly out of the box. Polish "l striked" is indeed somehow special (while it shouldn't - general consensus is that "ł" is NOT a very different letter than "l" etc.). Problem is: long time ago, some person from Unicode Consortium involved in UCA (Unicode Collation Algorithm) development made rather questionable decision to not treat "ł, Ł" as wariants of "l, L". It became a major PITA from than on - for example, it does also affect mysql utf8_general_ci & utf8_unicode_ci collations, to name just a few side effects. BTW, I believe this got finally corrected in subsequent UCA revisions (in v5.2.0+, AFAIRC), so this workaround may be no longer necessary in Koha installations with more recent libicu versions (?). -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10939 --- Comment #13 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- (In reply to Jacek Ablewicz from comment #12)
I think you only need those two lines in etc/words-icu.xml:
<transliterate rule="{ ł > l "/>
We are currently trying to improve this file in our library, and we use this solution (with other tricks) M. Saby -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10939 --- Comment #14 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- Fridolyn, we can have Polish records in a french or american catalog. So, why not improving the default etc/words-icu.xml for everybody? (same remarks for german umlauts, french ligatures etc) Mathieu -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10939 --- Comment #15 from Fridolyn SOMERS <fridolyn.somers@biblibre.com> --- (In reply to mathieu saby from comment #14)
So, why not improving the default etc/words-icu.xml for everybody? See comment 8. It may be rediscussed.
-- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10939 Julien Sicot <julien.sicot@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |julien.sicot@gmail.com --- Comment #16 from Julien Sicot <julien.sicot@gmail.com> --- For information, ICU provide already a package for processing similar transliteration. With just one line in etc/words-icu.xml (and etc/phrases-icu.xml) : <transform rule="[:Latin:] Latin-ASCII;"/> you can convert latin unicode to ascii. Normally, this works for all polish diacriticals. You can test icu rules here : http://demo.icu-project.org/icu-bin/translit or using yaz-icu. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10939 --- Comment #17 from Jacek Ablewicz <abl@biblos.pk.edu.pl> --- Ouch, looks like problems with "ł, Ł' are not related to UCA < v5.2 quirks like I thought previously; not at all. Real problem here being that: NFD; [:Nonspacing Mark:] Remove; NFC apparently does not work for 'striked through' latin characters (same problem with: ø, Ø, ƶ, Ƶ etc., not just with Ł, ł). I guess that's because NFD is not decomposing 'ł' to 'l + /' like it does for accented characters and so on. For ICU indexing to behave (more or less) like CHR (zebradb/etc/word-phrase-utf.chr) did, we can: 1) add something like that: <transliterate rule="{ Ø > o "/> <transliterate rule="{ ø > o "/> <transliterate rule="{ Đ > d "/> <transliterate rule="{ đ > d "/> <transliterate rule="{ Ħ > h "/> <transliterate rule="{ ħ > h "/> <transliterate rule="{ Ł > l "/> <transliterate rule="{ ł > l "/> <transliterate rule="{ Ŧ > t "/> <transliterate rule="{ ŧ > t "/> <transliterate rule="{ Ƶ > z "/> <transliterate rule="{ ƶ > z "/> <transliterate rule="{ Ǥ > g "/> <transliterate rule="{ ǥ > g "/> <transliterate rule="{ Ⱥ > a "/> <transliterate rule="{ ⱥ > a "/> <transliterate rule="{ Ȼ > c "/> <transliterate rule="{ ȼ > c "/> <transliterate rule="{ Ɇ > e "/> <transliterate rule="{ ɇ > e "/> <transliterate rule="{ Ɍ > r "/> <transliterate rule="{ ɍ > r "/> <transliterate rule="{ Ɏ > y "/> <transliterate rule="{ ɏ > y "/> <transliterate rule="{ Ɨ > i "/> <transliterate rule="{ ɨ > i "/> <transliterate rule="{ ʉ > u "/> <transliterate rule="{ Ʉ > u "/> <transliterate rule="{ Ӕ > ae "/> <transliterate rule="{ ӕ > ae "/> <transliterate rule="{ Œ > oe "/> <transliterate rule="{ œ > oe "/> to words-icu.xml, or: 2) as Julien suggested, we may use built-in Latin-ASCII ICU transliterator, i.e. add: <transform rule="[:Latin:] Latin-ASCII"/> Solution 2) looks much better IMO (it's more general-purpose) but it may not be ideal for everybody, as Latin-ASCII transliterator is not implemented in pre-4.6 ICU versions. -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org