locale dependencies and perl
Hi developers, just a thing I thought could be very interesting, please read this in a latin1 capable e-mail agent: Aparently marc8 records with latin1 extensions are being correctly stored in database, when marc8 is specified as source encoding, for z3950 records that come in that format. *** Well..., not so. *** Internet Explorer renders correctly the not so correct marc8 to utf8 transformation where chars from latin1 character repertoire are recognized in the transformation. What is really doing koha, when doing marc8 decoding/MARC21, is to store in database a sequence of: base character + unicode form of (tilde|cute|grave...etc) ...that is: a´, a`,n~, etc... ...with ` or ´ or etc... in UTF-8 (using 3 native bytes instead of 2 native bytes) Instead of: á, à, etc... (2 native bytes) Internet Explorer, not surprisingly for me, renders a´ as á, etc... but no other tools do it this way, for example firefox renders í as i with an upper to the dot acute. Ignacio Javier Gómez Rodríguez Analista - Programador Tfno: 902905590 - Fax: 981571425 ignacio.gomez@coremain.com www.coremain.com _______________________________________________ Koha-devel mailing list Koha-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/koha-devel
Hi, On Fri, Apr 11, 2008 at 4:26 AM, Ignacio Javier <ignacio.gomez@coremain.com> wrote:
What is really doing koha, when doing marc8 decoding/MARC21, is to store in database a sequence of:
base character + unicode form of (tilde|cute|grave...etc)
...that is:
a´, a`,n~, etc...
...with ` or ´ or etc... in UTF-8 (using 3 native bytes instead of 2 native bytes)
Instead of:
á, à, etc... (2 native bytes)
Internet Explorer, not surprisingly for me, renders a´ as á, etc... but no other tools do it this way, for example firefox renders í as i with an upper to the dot acute.
The UTF-8 is valid, it just may not be in the ideal normalization form. The strings that MARC::Charset produces when it converts from MARC-8 are in a decomposed Unicode normalization form, either NFD or NFKD. Some web browsers can render NFD strings without any difficulty, while other ones seem to work better if NFC is used. Right now Koha passes UTF-8 strings to the browser without renormalizing them, but perhaps we should be automatically converting them to NFC? Regards, Galen -- Galen Charlton Koha Application Developer LibLime galen.charlton@liblime.com p: 1-888-564-2457 x709 _______________________________________________ Koha-devel mailing list Koha-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/koha-devel
The UTF-8 is valid,
Properly, I think the 3 byte representation for a glyph (ñ) than in UTF-8 is two bytes does not fit in UTF-8 valid, but more in Unicode NFKD, isn't it?
it just may not be in the ideal normalization form. The strings that MARC::Charset produces when it converts from MARC-8 are in a decomposed Unicode normalization form, either NFD or NFKD. Some web browsers can render NFD strings without any difficulty, while other ones seem to work better if NFC is used. Right now Koha passes UTF-8 strings to the browser without renormalizing them, but perhaps we should be automatically converting them to NFC?
Has to be mantined in NFKD for compatibility with...? In that case shouldn't be wider audience to split all possibilities, given marc8 to utf-8 with unicode normalization in NFD, NFKD and NFKC, apart from the preferred option NFC? Ignacio Javier Gómez Rodríguez Analista - Programador Tfno: 902905590 - Fax: 981571425 ignacio.gomez@coremain.com www.coremain.com
Regards,
Galen -- Galen Charlton Koha Application Developer LibLime galen.charlton@liblime.com p: 1-888-564-2457 x709
_______________________________________________ Koha-devel mailing list Koha-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/koha-devel
participants (2)
-
Galen Charlton -
Ignacio Javier