Joshua Ferraro a écrit :
Hi Henri-Damien,
The problems you're describing are because your MARC data is not valid. You may have UTF-8 encoded data in your MARC, but unless the Leader 09 is set to 'a', the MARC::* suite of tools has no way of knowing it's UTF-8 and will interpret it as MARC-8 ( properly, since there are only two valid encodings in MARC21 ). MARC::* has no knowledge of any other encoding, but in the latest sourceforge version there is an optional UNIMARC flag you can pass to it that will avoid all character set conversions (but this needs to be tested as the author didn't have direct access to UNIMARC.
You can not simply switch from MARC-8 to UTF-8 without doing some heavy transformations ... for instance, in MARC-8, combining characters are of the form <combined character> <base> whereas in UTF-8, they are of the form <base> <combined character>.
If you are converting a database that has invalid MARC-8 data (for instance, if it has latin1 data) you'll want to use the ignore_errors flag:
MARC::Charset->ignore_errors(1);
to avoid losing entire subfields because of a bad character.
I have confirmed that a UTF-8 Koha is already possible, without changing CGI or DBI. So long as MySQL and Apache have been set up properly, UTF-8 data passes unharmed between the DBI and CGI levels. You can view an example of a pure UTF-8 Koha here: http://wipoopac.liblime.com
MARC::* is difficult to install properly, and it's also hard to make sure you have valid MARC records, but once you do these steps, the process works very smoothly. If we are to claim MARC21 compliance, MARC::* is a must (unless you want to write a new suite for MARC handling in perl or use a non-perl solution).
Thanks for your reply. Maybe I should use private message to report these thoughts. OK for claiming MARC compliance, as soon as it is for ANY MARC flavor would it be UNIMARC or else. But I am not OK to say pure UTF-8 Koha is already possible, since it is unfortunately not. I reported a display error and reported to How you can get these errors through a simple script. Both DBI and CGI are buggy in their UTF-8 management, even though it is true they donot harm UTF-8 data. But if PERL is to cope with utf8 data, it has to be aware of that and encode things properly. Maybe for you it works since you have no utf-8 *Both* in your zebra records *and* your framework or any mysqldata. But for us it IS a blocking problem and we HAVE to cope with it. As soon as we work only with mysql or only with zebra, no problems, as I said. But we are not. And we must be aware of that. But you gave me at least two pennies to think about : 1) LEADER HAS TO be well integrated (in MARChtml2xml in rel 3_0, it is not even generated) 2) MARC::Charset->ignore_errors can be used but is not the best solution since some data could be lost without notice. Many thanks anyway. -- Henri-Damien LAURENT