MARC character encoding
Hi koha-dev, and hi ed, (who is the author of MARC-Charset, and joined koha a few weeks ago) Could someone explain how to translate the "MARC21" charset to a more convenient one (and which is more convenient ?) Same question for UNIMARC (which is ISO646 if my docs are right) I tried MARC-Charset, which seems to translate from "MARC21" to UNICODE, but i don't know what to do with my unicode ;-( I tried : my $charset = MARC::Charset->new(); print $charset->to_utf8($unimarc->as_formatted())."\n"; where $unimarc is a MARC::Record containing a MARC21 record converted to UNIMARC (hope everybody understand this : i mapped marc21 fields to unimarc ones) -- Paul POULAIN Consultant indépendant en logiciels libres responsable francophone de koha (SIGB libre http://www.koha-fr.org)
Hi Paul: On Wed, Jan 15, 2003 at 03:20:47PM +0100, paul POULAIN wrote:
Could someone explain how to translate the "MARC21" charset to a more convenient one (and which is more convenient ?) Same question for UNIMARC (which is ISO646 if my docs are right)
If we lived in a perfect world we would all be using Unicode (UTF8) since it covers so many of the worlds scripts [1]. Unfortunately the world is not perfect. MARC has been around longer than Unicode, so MARC-8 character encoding to allow non Latin scripts to live in MARC records. I guess the world has bigger problems than character encodings (Mr George Bush comes to mind), but I'll leave that particular problem alone :) I wasn't aware that UNIMARC had defined a different standard for character encoding. Isn't ISO646 just an synonym for ASCII? [2] Which docs describe the character sets used in UNIMARC?
I tried MARC-Charset, which seems to translate from "MARC21" to UNICODE, but i don't know what to do with my unicode ;-(
Yes, MARC::Charset is an implementation of the MARC-8 ==> Unicode (UTF-8) mappings published by the Library of Congress. [3] In MARC-8 there is a special way of 'escaping' to other character sets (Hebrew, Cyrillic, East-Asian, etc).
I tried : my $charset = MARC::Charset->new(); print $charset->to_utf8($unimarc->as_formatted())."\n"; where $unimarc is a MARC::Record containing a MARC21 record converted to UNIMARC (hope everybody understand this : i mapped marc21 fields to unimarc ones)
The to_utf8() method will take a string of characters (encoded in MARC-8) and convert them to Unicdoe (UTF8). Initially I wanted to do this so that MARC records could be expressed as XML with UTF8 encoding. You mapped all the UNIMARC fields to MARC fields!?! I was under the impression that this was quite a big undertaking to do completely. Is your code currently checked into CVS? Having a UNIMARC filter in MARC::Record (MARC::File::UNIMARC) has been a long term goal. Maybe we could roll this work into the MARC::Record package? I'm not sure how you want to handle character encoding in Koha. Does MySQL properly handle Unicode (UTF8)? If it does I think long term Koha should probably attempt to store everything in UTF8. Luckily UTF8 is backwards compatible with plain vanilla ASCII. Even if MySQL handles storing UTF8, it would take some research to make sure DBD::mysql also does. Otherwise, there needs to be some global config option that determines Koha's character set encoding. There is the Encode [4] module (standard with 5.8.0) which could handle translating between a variety of character encodings. Hope this helps more than it hurts :) //Ed [1] http://www.unicode.org [2] http://czyborra.com/charsets/iso646.html [3] http://www.loc.gov/marc/specifications/specchartables.html [4] http://search.cpan.org/author/DANKOGAI/Encode-1.84/ -- % perl -MData::Dumper -e "print Dumper($me)" $VAR1 = { 'WEB' => 'http://www.inkdroid.org', 'NAME' => 'Ed Summers', 'AIM' => 'inkdroid', 'EMAIL' => 'ehs@pobox.com' };
In article <20030115162021.GB7111@dancygier.dhs.org>, Ed Summers <ehs@pobox.com> wrote: [...]
I'm not sure how you want to handle character encoding in Koha. Does MySQL properly handle Unicode (UTF8)? If it does I think long term Koha should probably attempt to store everything in UTF8. Luckily UTF8 is backwards compatible with plain vanilla ASCII. Even if MySQL handles storing UTF8, it would take some research to make sure DBD::mysql also does.
FYI, for certain languages (esp. CJK, esp. C which is a mess), UTF8 will be semi-mandatory to make basic functionality like searching work (the alternative being constructing complicated and MySQL-specific regexp in the SQL commands). In the future, I agree that storing UTF8 in the backend is the way to go.
Otherwise, there needs to be some global config option that determines Koha's character set encoding. There is the Encode [4] module (standard with 5.8.0) which could handle translating between a variety of character encodings.
I think Koha should have a global config option -- or perhaps it should have two: one for the backend, and one for the frontend. We might, e.g., allow the backend to use UTF8 but the frontend to use ISO-8859-1. This is not as far-fetched as it seems; e.g., if a library need to provide both an English and Maori interface (or French and Chinese) for Koha, this would make perfect sense (I think). -- Ambrose Li <a.c.li@ieee.org> http://ada.dhs.org/~acli/cmcc/ http://www.cccgt.org/ DRM is theft - We are the stakeholders
Ed Summers a écrit:
Hi Paul: On Wed, Jan 15, 2003 at 03:20:47PM +0100, paul POULAIN wrote:
Could someone explain how to translate the "MARC21" charset to a more convenient one (and which is more convenient ?) Same question for UNIMARC (which is ISO646 if my docs are right)
If we lived in a perfect world we would all be using Unicode (UTF8) since it covers so many of the worlds scripts [1]. Unfortunately the world is not perfect. MARC has been around longer than Unicode, so MARC-8 character encoding to allow non Latin scripts to live in MARC records. I guess the world has bigger problems than character encodings (Mr George Bush comes to mind), but I'll leave that particular problem alone :)
50% of your news here are related to Mr George Bush. Unfortunately for me, the other 50% are NOT related to character encoding :-)))
I wasn't aware that UNIMARC had defined a different standard for character encoding. Isn't ISO646 just an synonym for ASCII? [2] Which docs describe the character sets used in UNIMARC?
No, you're right. ISO646 IS Ascii. What i don't understand is how they code >127 codes on 2 digits. for example, \xc3\x65 = ê It's not ASCII ?
I tried MARC-Charset, which seems to translate from "MARC21" to UNICODE, but i don't know what to do with my unicode ;-(
Yes, MARC::Charset is an implementation of the MARC-8 ==> Unicode (UTF-8) mappings published by the Library of Congress. [3] In MARC-8 there is a special way of 'escaping' to other character sets (Hebrew, Cyrillic, East-Asian, etc).
this special way is \xc3\x65 ?
You mapped all the UNIMARC fields to MARC fields!?! I was under the impression that this was quite a big undertaking to do completely. Is your code currently checked into CVS? Having a UNIMARC filter in MARC::Record (MARC::File::UNIMARC) has been a long term goal. Maybe we could roll this work into the MARC::Record package?
NO, of course. You're right, this is a BIG job. I did this only for a few fields/subfields I needed (around 20-25 fields) It's a 10-20 lines script (+ the mapping array) with MARC::Record - Paul POULAIN Consultant indépendant en logiciels libres responsable francophone de koha (SIGB libre http://www.koha-fr.org)
Ed Summers a écrit:
What i don't understand is how they code >127 codes on 2 digits. for example, \xc3\x65 = ê It's not ASCII ?
It is not MARC-8, and it is not ISO-8859-1, and i'm not sure what it is. //Ed
Francois lemarchand sended me a little script to translate éà... into 8859-1 standard characters. I've included it in the addbiblio.pl script (when the system finds a biblio in the breeding farm) It seems to work. Things are definetly strange in char encoding. Uploaded in cvs a few minuts ago -- Paul POULAIN Consultant indépendant en logiciels libres responsable francophone de koha (SIGB libre http://www.koha-fr.org)
On Tue, Jan 21, 2003 at 09:15:07AM +0100, paul POULAIN wrote:
Francois lemarchand sended me a little script to translate éà... into 8859-1 standard characters. I've included it in the addbiblio.pl script (when the system finds a biblio in the breeding farm) It seems to work. Things are definetly strange in char encoding.
Uploaded in cvs a few minuts ago
I'm looking at the script. From the comments it looks like Francois' code is converting from ISO 5426 to ISO 8859-1. How are character sets handled in UNIMARC? I'm guessing there are more character sets than ISO 5426 which can be used. I just checked and Perl's Encode::* modules don't seem to handle ISO 5426 :( which is a shame. It is even more a shame that ISO doesn't make these standards public. I'm going to subscribe to perl-unicode@perl.org and see if I can find out more. //Ed -- Ed Summers aim: inkdroid web: http://www.inkdroid.org
Ed Summers a écrit:
On Tue, Jan 21, 2003 at 09:15:07AM +0100, paul POULAIN wrote:
Francois lemarchand sended me a little script to translate éà... into 8859-1 standard characters. I've included it in the addbiblio.pl script (when the system finds a biblio in the breeding farm) It seems to work. Things are definetly strange in char encoding.
Uploaded in cvs a few minuts ago
I'm looking at the script. From the comments it looks like Francois' code is converting from ISO 5426 to ISO 8859-1. How are character sets handled in UNIMARC? I'm guessing there are more character sets than ISO 5426 which can be used.
I just checked and Perl's Encode::* modules don't seem to handle ISO 5426 :( which is a shame. It is even more a shame that ISO doesn't make these standards public. I'm going to subscribe to perl-unicode@perl.org and see if I can find out more.
//Ed
Sorry, but i've more deeply looked at francois code, and some MARC21 and UNIMARC files. My conclusion is that the following code : s/\xe1/\xc1/gm; s/\xe2/\xc2/gm; s/\xe3/\xc3/gm; s/\xe4/\xc4/gm; s/\xe8/\xc8/gm; s/\xe9/\xc9/gm; s/\xf0/\xd0/gm; is enough to migrate from MARC21 to UNIMARC char coding. It tried this on my marc21->unimarc script, on 30 000 records, and it works fine. So, i think we have 2 complete tables (marc21 and unimarc) in Biblio.pm, that i commited a few minuts ago. -- Paul POULAIN Consultant indépendant en logiciels libres responsable francophone de koha (SIGB libre http://www.koha-fr.org)
My conclusion is that the following code : s/\xe1/\xc1/gm; s/\xe2/\xc2/gm; s/\xe3/\xc3/gm; s/\xe4/\xc4/gm; s/\xe8/\xc8/gm; s/\xe9/\xc9/gm; s/\xf0/\xd0/gm; is enough to migrate from MARC21 to UNIMARC char coding. It tried this on my marc21->unimarc script, on 30 000 records, and it works fine.
So, i think we have 2 complete tables (marc21 and unimarc) in Biblio.pm, that i commited a few minuts ago.
This is pretty sketchy, and I would guess far from complete. Perhaps it's OK for your purposes but I wouldn't be comfortable sending my data through it. In the absence of any documentation on UNIMARC character encoding I guess you've decided that it will have to do. //Ed
Ed Summers a écrit: It's trully a pain... In MARC21, the "à" is coded \xe1\x61. In UNIMARC, it's coded... \xc1\x61 And that's the same for all "escaped" characters :-(((((((( Worst than this... The \xe1 character in UNIMARC means... Æ, so it's possible to find \xe1\x61 in UNIMARC, it means Æa It's really a nightmare to manage !!! I end with a question : is there in a MARC record a way to know wether it's unimarc, marc21, or another marc flavor. I've found nothing in the marc leader, and don't think there is one (which is definetly bad thing if i'm right) -- Paul POULAIN Consultant indépendant en logiciels libres responsable francophone de koha (SIGB libre http://www.koha-fr.org)
I end with a question : is there in a MARC record a way to know wether it's unimarc, marc21, or another marc flavor. I've found nothing in the marc leader, and don't think there is one (which is definetly bad thing if i'm right)
As far as I know there is no way. Is there a good online reference for UNIMARC similar to MARC21 [1]? If there is I'd be willing to look into it a bit further. //Ed [1] http://lcweb.loc.gov/marc -- % perl -MData::Dumper -e "print Dumper($me)" $VAR1 = { 'WEB' => 'http://www.inkdroid.org', 'NAME' => 'Ed Summers', 'AIM' => 'inkdroid', 'EMAIL' => 'ehs@pobox.com' };
Ed Summers a écrit:
I end with a question : is there in a MARC record a way to know wether it's unimarc, marc21, or another marc flavor. I've found nothing in the marc leader, and don't think there is one (which is definetly bad thing if i'm right)
As far as I know there is no way. Is there a good online reference for UNIMARC similar to MARC21 [1]? If there is I'd be willing to look into it a bit further.
I use a paper UNIMARC reference book. I've deeply searched, and it really seems to be nothing to find which MARC flavour it is. Let's hope Biblio-XML will arrive soon :-) -- Paul POULAIN Consultant indépendant en logiciels libres responsable francophone de koha (SIGB libre http://www.koha-fr.org)
participants (3)
-
acli＠ada.dhs.org via news-to-mail gateway -
Ed Summers -
paul POULAIN