Hi all, I've spent the better part of two days working on the encoding problems that several of the Koha developers have been experiencing and I think I've met with some success. First, software I'm running: Perl 5.8.4 XML::SAX 0.14 MARC::Record 2.0RC1 MARC::File::XML 0.83 MARC::Charset 0.95 There were two problems ... the first happens when you have MARC21 records with wrongly encoded characters. Let me be clear that I mean wrongly encoded by the MARC21 MARC-8 encoding. The LOC has defined all of the valid MARC-8 characters and mapped them to UTF-8 here: http://www.loc.gov/marc/specifications/codetables.xml (warning, very large file) This is the code table that MARC::Charset uses to convert MARC-8 encoded records into UTF-8 ... and that's what MARC::File::XML uses to convert a binary MARC file into MARCXML. By default, if MARC::Charset encounters a character that isn't in that code table, it drops the whole subfield and throws a warn like this: no mapping found at position 8 in Price : � 7.99; Inv.# B 476913; Date 06/03/98; Supplier : Dawson UK; Recd 20/03/98; Contents : 1. The problem : 1. Don't bargain over positions; 2. The method : 2. Separate the people from the problem; 3. Focus on interests, not positions; 4. Invent options for mutual gain; 5. Insist on using objective criteria; 3. Yes, but : 6. What if they are more powerful? 7. What if they won't play? 8. What if they use dirty tricks? 4. In conclusion; 5. Ten questions people ask about getting to yes; g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/local/share/perl/5.8.4/MARC/Charset.pm line 197. I don't know if you can see it or not, but before the 7.99 in the above dump is a \x9C character, which is an invalid MARC-8 character. The temporary solution is to add the following: use MARC::Charset; MARC::Charset->ignore_errors(1); to any script or module that you expect to encounter wrong encodings. This way, it will just drop the offending character rather than the whole subfield (it still throws the warn though). I'm not 100% happy with that solution, but I can deal with it until someone has better suggestion. The second problem is the more serious one ... several of us have tried to pass UTF-8 encoded XML records in to the new_from_xml() method and had the parser crash if there were 'combining characters' in the record. The problem seems to be with the PurePerl parser ... as soon as I installed XML::SAX::Expat it went away. I've attached a script to this email that you can use to test your system to make sure things are set up correctly. The script attempts to convert a binary MARC record in either UTF-8 or MARC-8 encoding, to MARC-XML (encoded as UTF-8) and then back to binary MARC (as UTF-8). You can test the first problem with the following record: http://liblime.com/public/badencoding.mrc And the second with this one: http://liblime.com/public/combiningchar.mrc Run the script like this: $ ./roundtrip.pl badencoding.mrc out.utf8.mrc dump Compare the original MARC record with the out.utf8.mrc ... the only difference should be in the encoding and possibly missing chars if your records had bad encoding. (you can edit roundtrip.pl to turn on/off the ignore_errors flag). And don't forget to install XML::SAX::Expat. So ... now that we've got the MARC21 encoding problems out of the way, we need to look at UNIMARC. Mike Rylander has already done some work on this but I think due to the second error above, we've not had success testing thusfar. Cheers, -- Joshua Ferraro VENDOR SERVICES FOR OPEN-SOURCE SOFTWARE President, Technology migration, training, maintenance, support LibLime Featuring Koha Open-Source ILS jmf@liblime.com |Full Demos at http://liblime.com/koha |1(888)KohaILS