UTF-8 problems : a summary and some solutions
Hi, I investigated quite hard into finding a solution to our utf-8 problems. It seems we CAN come to a solution. What is the problem ? ------------------------------ On my right : values from xml records utf-8 encoded. Correctly displaying them is only a matter of making PERL aware of the fact that he is using utf-8 values, that is : - either use perl -CS or perl -CIOE instead of just simple perl on the first line of the script. That toggles perl into utf-8 for STDINPUT, STDOUTPUT and STDERR. - or add binmode STDOUT, ":UTF-8" or binmore STDOUT, ":utf8" (Be warned that those two different descriptions ARE NOT equivalent. see http://perldoc.perl.org/Encode.html#UTF-8-vs.-utf8 I would suggest that we always use utf8 rather than UTF-8) Simple, clear, efficient (remember though to use libxml2 as stated Joshua in a previous mail http://www.nntp.perl.org/group/perl.perl4lib/2369). On my left : values from Mysql coded in utf-8 and stored as such in MysqlDatabase. When using ("SET names=UTF-8") on the database just after the connection, Everything goes... WELL !!!! Bibliothèque ¥ · £ · € · $ · ¢ · ₡ · ₢ · ₣ · ₤ · ₥ · ₦ · ₧ · ₨ · ₩ · ₪ · ₫ · ₭ · ₮ · So What is the problem ???? The problem is that left goes well PROVIDED that you DONOT set binmode STDOUT,":utf8" and PERL believes he is processing ISO-8859-1 data. If you launch test_mix.pl on your system without perl -CS : xml data wil be badly displayed BUT Mysql data Well displayed !!! <datafield tag="200" ind1=" " ind2=" "> <subfield code="a">(L') �ole des p�es</subfield> <subfield code="b">R</subfield> <subfield code="f">Bazin, Herv�/subfield> </datafield> <datafield tag="210" ind1=" " ind2=" "> <subfield code="c">�itions du seuil</subfield> </datafield> If you launch test_mix.pl on your system with perl -CS : xml data will be well displayed BUT Mysql data badly (encoded TWICE) !!! Bibliothèque Â¥ · £ · ⬠· $ · ¢ · ⡠· ⢠· ⣠· ⤠· ⥠· ⦠· ⧠· ⨠· ⩠· ⪠· ⫠· â · ⮠· OK. we've got a problem. Where does it come from ? It comes from DBD::mysql and DBI which provides utf-8 data to PERL without Telling him. In perl, Utf-8 data has a flag that tells OK, I am utf-8. If this flag is not set and output is UTF-8 Perl will "magically" but unfortunately re-encode the values. See http://www.mail-archive.com/dbi-dev@perl.org/msg04319.html and following... Solutions. ------------- you can have a look at http://www.simplicidade.org/notes/archives/2005/12/utf8_and_dbdmys.html There are three solutions : - find out EVERYWHERE in the code we receive data from mysql and set the flag on |if (! utf8::is_utf8($message)) { utf8::decode($message); }| Is there a volunteer ? ;) - patch DBD-MYSQL code so that it makes perl utf-8 aware and use {*"*mysql_enable_utf8*"*=>1} options when connecting to database. There is a patch here (version 2.90008 http://lists.mysql.com/perl/3563 and here http://rt.cpan.org/Public/Bug/Display.html?id=17829 version 3.000006 ) And it seems that people in the DBD-MYSQL team will incorporate this patch in next DBD::mysql version. - Use (http://dysphoria.net/2006/02/05/utf-8-a-go-go/) UTF8DBI.pm for any database connections rather than DBD-mysql. Advantage : If we incorporate those package tightly to C4 and Koha, we may improve UTF-8 compliance and stability. Disadvantages : Module is not supported by anyone, not official, and would require quite HEAVY transformations in our code. You may have felt which solution seems to me the most reasonale one. The only problem being... Are we finished with utf-8 problems ? I HOPE So. But we can reasonably think there still may be hidden ones. Why ? - http://www.mhonarc.org/archive/html/perl-unicode/2006-07/msg00004.html and http://www.mhonarc.org/archive/html/perl-unicode/2006-07/msg00008.html seems to point out some CGI input problems with UTF-8. (It is a fact I DIDNOT test data input with binmode UTF-8 set or with the DBD-MYSQL hack.) So I go on testing. If you need test files, you can get them on http://hdlaurent.paulpoulain.com/test_mix.pl http://hdlaurent.paulpoulain.com/test_mix_fixed.pl http://hdlaurent.paulpoulain.com/testsrecord.xml -- Henri-Damien LAURENT
Just following this entry. I am still working on utf-8 compliance... And I am now wondering if the use of MARC::File::XML and MARC::Charset is a good solution. Indeed, while trying to add a new biblio, using the solution I explained in the previous message. I face two problems : First : CGI is not utf-8 "aware". That is to say that if you provide CGI utf-8 text as input, and perl is told to use utf-8 as input data, CGI will not place utf-8 flag onto the data provided, so that it will be double encoded by perl ! Solution : Not using UTF-8 on standard input. It will be OK. Maybe another patch would be helpful. But I find only the one I told on the previous message. Or decoding any entry to put the right flags on ? Second : when trying to input TRUE utf-8 data, namely : title : mémé est la plus forte..... ∮ E⋅da = Q, n → ∞, ∑ f(i) = ∏ g(i), ∀x∈ℝ: ⌈x⌉ = −⌊−x⌋, α ∧ ¬β = ¬(¬α ∨ β), publisher : test(i), ∀x∈ℝ: ⌈x and publication year : (i), ∀x∈ℝ: ⌈x to addbiblio.pl, MARChtml2xml works and I get : xml : <?xml version="1.0" encoding="UTF-8"?> <collection xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.loc.gov/MARC21/slim http://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd" xmlns="http://www.loc.gov/MARC21/slim"> <datafield tag="101" ind1="" ind2=""> <subfield code="a">spa</subfield> </datafield> <datafield tag="200" ind1="" ind2=""> <subfield code="a">m\xc3\xa9m\xc3\xa9 est la plus forte..... \xe2\x88\xae E\xe2\x8b\x85da = Q, n \xe2\x86\x92 \xe2\x88\x9e, \xe2\x88\x91 f(i) = \xe2\x88\x8f g(i), \xe2\x88\x80x\xe2\x88\x88\xe2\x84\x9d: \xe2\x8c\x88x\xe2\x8c\x89 = \xe2\x88\x92\xe2\x8c\x8a\xe2\x88\x92x\xe2\x8c\x8b, \xce\xb1 \xe2\x88\xa7 \xc2\xac\xce\xb2 = \xc2\xac(\xc2\xac\xce\xb1 \xe2\x88\xa8 \xce\xb2),</subfield> </datafield> <datafield tag="210" ind1="" ind2=""> <subfield code="c">test(i), \xe2\x88\x80x\xe2\x88\x88\xe2\x84\x9d: \xe2\x8c\x88x</subfield> <subfield code="d">(i), \xe2\x88\x80x\xe2\x88\x88\xe2\x84\x9d: \xe2\x8c\x88x</subfield> </datafield> </collection> But new biblio is failing. it get these errors : <http://localhost/dotclear/ecrire/Tue%20Aug%2022%2012:03:35%202006>no mapping found at position 37 in m\xc3\xa9m\xc3\xa9 est la plus forte..... \xe2\x88\xae E\xe2\x8b\x85da = Q, n \xe2\x86\x92 \xe2\x88\x9e, \xe2\x88\x91 f(i) = \xe2\x88\x8f g(i), \xe2\x88\x80x\xe2\x88\x88\xe2\x84\x9d: \xe2\x8c\x88x\xe2\x8c\x89 = \xe2\x88\x92\xe2\x8c\x8a\xe2\x88\x92x\xe2\x8c\x8b, \xce\xb1 \xe2\x88\xa7 \xc2\xac\xce\xb2 = \xc2\xac(\xc2\xac\xce\xb1 \xe2\x88\xa8 \xce\xb2), g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/lib/perl5/site_perl/5.8.7/MARC/Charset.pm line 188. <http://localhost/dotclear/ecrire/Tue%20Aug%2022%2012:03:35%202006>no mapping found at position 11 in test(i), \xe2\x88\x80x\xe2\x88\x88\xe2\x84\x9d: \xe2\x8c\x88x g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/lib/perl5/site_perl/5.8.7/MARC/Charset.pm line 188. <http://localhost/dotclear/ecrire/Tue%20Aug%2022%2012:03:35%202006>no mapping found at position 7 in (i), \xe2\x88\x80x\xe2\x88\x88\xe2\x84\x9d: \xe2\x8c\x88x g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/lib/perl5/site_perl/5.8.7/MARC/Charset.pm line 188. Then add fails. So I think that comes from MARC::File::XML that uses MARC::Charset to try and get the data from the XML, and MARC::Charset tries to decode the utf-8 data as if it was ASCII and it is not. And then spoils the data. So my question is : Do we really need to use MARC::File::XML as such or can we hack it so that data would be taken as utf-8 ? Should we hack MARC::Charset ? Should we hack CGI or create a new package to mark utf-8 data as utf-8 and convert non-utf-8 data to utf-8 ? -- Henri-Damien LAURENT
Hi Henri-Damien, On Wed, Aug 23, 2006 at 10:35:18AM +0200, Henri-Damien LAURENT wrote:
Just following this entry. I am still working on utf-8 compliance... And I am now wondering if the use of MARC::File::XML and MARC::Charset is a good solution.
Indeed, while trying to add a new biblio, using the solution I explained in the previous message. I face two problems : First : CGI is not utf-8 "aware". That is to say that if you provide CGI utf-8 text as input, and perl is told to use utf-8 as input data, CGI will not place utf-8 flag onto the data provided, so that it will be double encoded by perl ! Solution : Not using UTF-8 on standard input. It will be OK. Maybe another patch would be helpful. But I find only the one I told on the previous message. Or decoding any entry to put the right flags on ?
Second : when trying to input TRUE utf-8 data, namely : title : mémé est la plus forte..... ∮ E⋅da = Q, n → ∞, ∑ f(i) = ∏ g(i), ∀x∈ℝ: ⌈x⌉ = −⌊−x⌋, α ∧ ¬β = ¬(¬α ∨ β), publisher : test(i), ∀x∈ℝ: ⌈x and publication year : (i), ∀x∈ℝ: ⌈x
to addbiblio.pl, MARChtml2xml works and I get : xml : <?xml version="1.0" encoding="UTF-8"?> <collection xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.loc.gov/MARC21/slim http://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd" xmlns="http://www.loc.gov/MARC21/slim"> <datafield tag="101" ind1="" ind2=""> <subfield code="a">spa</subfield> </datafield> <datafield tag="200" ind1="" ind2=""> <subfield code="a">m\xc3\xa9m\xc3\xa9 est la plus forte..... \xe2\x88\xae E\xe2\x8b\x85da = Q, n \xe2\x86\x92 \xe2\x88\x9e, \xe2\x88\x91 f(i) = \xe2\x88\x8f g(i), \xe2\x88\x80x\xe2\x88\x88\xe2\x84\x9d: \xe2\x8c\x88x\xe2\x8c\x89 = \xe2\x88\x92\xe2\x8c\x8a\xe2\x88\x92x\xe2\x8c\x8b, \xce\xb1 \xe2\x88\xa7 \xc2\xac\xce\xb2 = \xc2\xac(\xc2\xac\xce\xb1 \xe2\x88\xa8 \xce\xb2),</subfield> </datafield> <datafield tag="210" ind1="" ind2=""> <subfield code="c">test(i), \xe2\x88\x80x\xe2\x88\x88\xe2\x84\x9d: \xe2\x8c\x88x</subfield> <subfield code="d">(i), \xe2\x88\x80x\xe2\x88\x88\xe2\x84\x9d: \xe2\x8c\x88x</subfield> </datafield> </collection> But new biblio is failing. it get these errors :
<http://localhost/dotclear/ecrire/Tue%20Aug%2022%2012:03:35%202006>no mapping found at position 37 in m\xc3\xa9m\xc3\xa9 est la plus forte..... \xe2\x88\xae E\xe2\x8b\x85da = Q, n \xe2\x86\x92 \xe2\x88\x9e, \xe2\x88\x91 f(i) = \xe2\x88\x8f g(i), \xe2\x88\x80x\xe2\x88\x88\xe2\x84\x9d: \xe2\x8c\x88x\xe2\x8c\x89 = \xe2\x88\x92\xe2\x8c\x8a\xe2\x88\x92x\xe2\x8c\x8b, \xce\xb1 \xe2\x88\xa7 \xc2\xac\xce\xb2 = \xc2\xac(\xc2\xac\xce\xb1 \xe2\x88\xa8 \xce\xb2), g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/lib/perl5/site_perl/5.8.7/MARC/Charset.pm line 188.
<http://localhost/dotclear/ecrire/Tue%20Aug%2022%2012:03:35%202006>no mapping found at position 11 in test(i), \xe2\x88\x80x\xe2\x88\x88\xe2\x84\x9d: \xe2\x8c\x88x g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/lib/perl5/site_perl/5.8.7/MARC/Charset.pm line 188.
<http://localhost/dotclear/ecrire/Tue%20Aug%2022%2012:03:35%202006>no mapping found at position 7 in (i), \xe2\x88\x80x\xe2\x88\x88\xe2\x84\x9d: \xe2\x8c\x88x g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/lib/perl5/site_perl/5.8.7/MARC/Charset.pm line 188.
Then add fails. So I think that comes from MARC::File::XML that uses MARC::Charset to try and get the data from the XML, and MARC::Charset tries to decode the utf-8 data as if it was ASCII and it is not. And then spoils the data. So my question is : Do we really need to use MARC::File::XML as such or can we hack it so that data would be taken as utf-8 ?
Should we hack MARC::Charset ? Should we hack CGI or create a new package to mark utf-8 data as utf-8 and convert non-utf-8 data to utf-8 ?
The problems you're describing are because your MARC data is not valid. You may have UTF-8 encoded data in your MARC, but unless the Leader 09 is set to 'a', the MARC::* suite of tools has no way of knowing it's UTF-8 and will interpret it as MARC-8 ( properly, since there are only two valid encodings in MARC21 ). MARC::* has no knowledge of any other encoding, but in the latest sourceforge version there is an optional UNIMARC flag you can pass to it that will avoid all character set conversions (but this needs to be tested as the author didn't have direct access to UNIMARC. You can not simply switch from MARC-8 to UTF-8 without doing some heavy transformations ... for instance, in MARC-8, combining characters are of the form <combined character> <base> whereas in UTF-8, they are of the form <base> <combined character>. If you are converting a database that has invalid MARC-8 data (for instance, if it has latin1 data) you'll want to use the ignore_errors flag: MARC::Charset->ignore_errors(1); to avoid losing entire subfields because of a bad character. I have confirmed that a UTF-8 Koha is already possible, without changing CGI or DBI. So long as MySQL and Apache have been set up properly, UTF-8 data passes unharmed between the DBI and CGI levels. You can view an example of a pure UTF-8 Koha here: http://wipoopac.liblime.com MARC::* is difficult to install properly, and it's also hard to make sure you have valid MARC records, but once you do these steps, the process works very smoothly. If we are to claim MARC21 compliance, MARC::* is a must (unless you want to write a new suite for MARC handling in perl or use a non-perl solution). Cheers, -- Joshua Ferraro SUPPORT FOR OPEN-SOURCE SOFTWARE President, Technology migration, training, maintenance, support LibLime Featuring Koha Open-Source ILS jmf@liblime.com |Full Demos at http://liblime.com/koha |1(888)KohaILS
Joshua Ferraro a écrit :
Hi Henri-Damien,
The problems you're describing are because your MARC data is not valid. You may have UTF-8 encoded data in your MARC, but unless the Leader 09 is set to 'a', the MARC::* suite of tools has no way of knowing it's UTF-8 and will interpret it as MARC-8 ( properly, since there are only two valid encodings in MARC21 ). MARC::* has no knowledge of any other encoding, but in the latest sourceforge version there is an optional UNIMARC flag you can pass to it that will avoid all character set conversions (but this needs to be tested as the author didn't have direct access to UNIMARC.
You can not simply switch from MARC-8 to UTF-8 without doing some heavy transformations ... for instance, in MARC-8, combining characters are of the form <combined character> <base> whereas in UTF-8, they are of the form <base> <combined character>.
If you are converting a database that has invalid MARC-8 data (for instance, if it has latin1 data) you'll want to use the ignore_errors flag:
MARC::Charset->ignore_errors(1);
to avoid losing entire subfields because of a bad character.
I have confirmed that a UTF-8 Koha is already possible, without changing CGI or DBI. So long as MySQL and Apache have been set up properly, UTF-8 data passes unharmed between the DBI and CGI levels. You can view an example of a pure UTF-8 Koha here: http://wipoopac.liblime.com
MARC::* is difficult to install properly, and it's also hard to make sure you have valid MARC records, but once you do these steps, the process works very smoothly. If we are to claim MARC21 compliance, MARC::* is a must (unless you want to write a new suite for MARC handling in perl or use a non-perl solution).
Thanks for your reply. Maybe I should use private message to report these thoughts. OK for claiming MARC compliance, as soon as it is for ANY MARC flavor would it be UNIMARC or else. But I am not OK to say pure UTF-8 Koha is already possible, since it is unfortunately not. I reported a display error and reported to How you can get these errors through a simple script. Both DBI and CGI are buggy in their UTF-8 management, even though it is true they donot harm UTF-8 data. But if PERL is to cope with utf8 data, it has to be aware of that and encode things properly. Maybe for you it works since you have no utf-8 *Both* in your zebra records *and* your framework or any mysqldata. But for us it IS a blocking problem and we HAVE to cope with it. As soon as we work only with mysql or only with zebra, no problems, as I said. But we are not. And we must be aware of that. But you gave me at least two pennies to think about : 1) LEADER HAS TO be well integrated (in MARChtml2xml in rel 3_0, it is not even generated) 2) MARC::Charset->ignore_errors can be used but is not the best solution since some data could be lost without notice. Many thanks anyway. -- Henri-Damien LAURENT
On Wed, Aug 23, 2006 at 03:44:20PM +0200, Henri-Damien LAURENT wrote:
OK for claiming MARC compliance, as soon as it is for ANY MARC flavor would it be UNIMARC or else. Not sure I understand what you mean ...
But I am not OK to say pure UTF-8 Koha is already possible, since it is unfortunately not. I reported a display error and reported to How you can get these errors through a simple script.
Both DBI and CGI are buggy in their UTF-8 management, even though it is true they donot harm UTF-8 data. But if PERL is to cope with utf8 data, it has to be aware of that and encode things properly. Maybe for you it works since you have no utf-8 *Both* in your zebra records *and* your framework or any mysqldata. But for us it IS a blocking problem and we HAVE to cope with it. As soon as we work only with mysql or only with zebra, no problems, as I said. But we are not.
And we must be aware of that. But you gave me at least two pennies to think about : 1) LEADER HAS TO be well integrated (in MARChtml2xml in rel 3_0, it is not even generated) 2) MARC::Charset->ignore_errors can be used but is not the best solution since some data could be lost without notice. The only data that is lost is the one invalid character that causes
http://wipoopac.liblime.com is pure UTF-8, Apache, MySQL database, table, column, etc., and it works just fine. The MARC data was imported using MARC::* suite, converted from MARC-8 encoding to UTF-8 encoding. What specific problems are you having? the problem. For instance, if you are using MARC21 records, and your record doesn't have a leader (or the leader claims to be MARC-8 encoded), and you have an invalid latin1 character in the data, it will lose that one character. But that character should not have been there in the first place! The third thing to be aware of is the UNIMARC flag that you can pass to MARC::File::XML ... for some reason, I've mentioned this about 10 times to UNIMARC guys but so far, noone has tested it! Cheers, -- Joshua Ferraro SUPPORT FOR OPEN-SOURCE SOFTWARE President, Technology migration, training, maintenance, support LibLime Featuring Koha Open-Source ILS jmf@liblime.com |Full Demos at http://liblime.com/koha |1(888)KohaILS
participants (2)
-
Henri-Damien LAURENT -
Joshua Ferraro