Gentlemen, I have a problem using UTF-8 with Koha. If I import MARC-Data via z3950 special characters are treated differently ÄÜÖäüö works at some pages but not in all. ß dosn't work at any page. This is the ü I get from z3950 and this is the ü which is generated when I type it. These are two different characters. I'm using Koha 2.2.8, MySQL5 and Apache 1.3 on Mac OSX 10.4 with Firefox. I've read the notes on the Wiki and my server, Database and the tables are set to UTF-8. I also set the templates encoding variable and added the encoding to httpd.conf The z3950 server sends data UTF-8 encoded Did I miss something? Dorian Meid
TARGET ISSUES. Are you certain that your Z39.50 target is returning records with UTF-8 encoding? If you supply the connection parameters and a test search for records which you believe are problematic, then I can test the target myself. BYTE CODES NEEDED FOR A RELIABLE CHECK. You need to compare the hexadecimal byte codes in the actual records with your expectation. Placing example characters in an email message is not enough because one has to separate problems of browsers not sending UTF-8 requests and systems rendering UTF-8 fonts correctly. Sadly, Firefox is a poor performer for transmitting data in UTF-8. I use Firefox regularly but it does have this data transmission problem when the client is in a non-UTF-8 environment. Some fonts used in Koha stylesheets are especially problematic for rendering characters correctly in some environments. The stylesheet font problem should have been fixed long ago but different developers have different environments and do not see the same problems for font rendering. (Please note that I do not run my email client in UTF-8 mode.) Thomas Dukleth Agogme 109 E 9th Street, 3D New York, NY 10003 USA http://www.agogme.com 212-674-3783 On Sun, March 11, 2007 12:18 am, Dorian Meid wrote:
Gentlemen,
I have a problem using UTF-8 with Koha. If I import MARC-Data via z3950 special characters are treated differently
ÄÜÖäüö works at some pages but not in all. ß dosn't work at any page.
This is the ü I get from z3950 and this is the ü which is generated when I type it. These are two different characters.
I'm using Koha 2.2.8, MySQL5 and Apache 1.3 on Mac OSX 10.4 with Firefox. I've read the notes on the Wiki and my server, Database and the tables are set to UTF-8. I also set the templates encoding variable and added the encoding to httpd.conf The z3950 server sends data UTF-8 encoded
Did I miss something?
Dorian Meid
_______________________________________________ Koha-devel mailing list Koha-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/koha-devel
Am 12.03.2007 um 03:19 schrieb Thomas Dukleth:
TARGET ISSUES.
Are you certain that your Z39.50 target is returning records with UTF-8 encoding? If you supply the connection parameters and a test search for records which you believe are problematic, then I can test the target myself.
BYTE CODES NEEDED FOR A RELIABLE CHECK.
Because I don't have another z39.50 client I use the scripts we have, namely z3950/search.pl I simply store the retreived marc data with: open(MARCFILE, ">:raw", "result$i.MARC"); print MARCFILE $marcdata; close MARCFILE; starting at line 174. I also tried: open(MARCFILE, ">:utf8", "result$i.MARC"); when I use a server which claims to send utf-8 i get: with raw-print: 75 CC 88 C3 with utf8-print: 00 75 03 08 FF FD both should be "üß" (ü ß) when I use a server which uses autonegotiation of the charset I get a correct encoded latin-1 record the ü is ok but FF FD for ß is definitely wrong. It seems to me that this encoding thing is a real pain and I'm rather new to it. In this example I used the host "z3950.gbv.de" port "20012" database "gvk" user "999" password "abc". I searched for the ISBN "3-552-06027-8" The title should be "Die Süße des Lebens" Another strange point is that the field ends at the ß character ("Die SüÃ"), but the original title is much longer (maybe indicates a faulty record, but it's the same with all records containing ß) Unfortunately I don't know any other server, which provides utf8 encoded MARC21 data. Maybe somebody can tell me one and a sample search with the results to expect.
Sadly, Firefox is a poor performer for transmitting data in UTF-8. Yep, I checked my browser here: http://www.fileformat.info/info/ unicode/utf8test.htm
Dorian Meid
Comments inline. On Mon, March 12, 2007 4:26 am, Dorian Meid wrote:
[snip]
In this example I used the host "z3950.gbv.de" port "20012" database "gvk" user "999" password "abc". I searched for the ISBN "3-552-06027-8" The title should be "Die Süße des Lebens"
I have tested and found that the problem is with GBV at least for this record. I had never tested their records retuned in UTF-8 from port 20012 previously. I only tested for the record you provided just now but found that all their other encodings served over various ports seem to be correct for this record. In my extensive previous testing of GBV I had only tested the MARC-8 encodings returned from port 20011. I had occasionally seen problems of encoding with individual records which had not been correctly encoded in MARC-8 but that may be unavoidable from any large data source, especially union catalogues like GBV. Koha does not yet have support for the other encodings they provide. I tested with my own client, not yet ported to Koha, which supports all the encodings I have found in use anywhere except for UNIMARC records from Greece. When different encodings are available as options from a particular target catalogue I have found that using the encoding most likely to have used in the original records is probably safest. In the case of GBV, that would be ISO 5426. Koha has no support for ISO 5426 yet unless it was added when I was not looking. Your best option for use with Koha is to use the MARC-8 encodings from port 20011. Not enough systems use UTF-8 records, therefore, UTF-8 services are often not well tested at many libraries. Even the Library of Congress, when returning records in UTF-8 will frequently have improper encodings so that MARC-8 is also the best option there. Please report this problem to GBV if you have any good contacts there.
[snip]
Thomas Dukleth Agogme 109 E 9th Street, 3D New York, NY 10003 USA http://www.agogme.com 212-674-3783
Thanks for validating this! Am 13.03.2007 um 12:14 schrieb Thomas Dukleth: ...
I have tested and found that the problem is with GBV at least for this record. I had never tested their records retuned in UTF-8 from port 20012 previously. ... Please report this problem to GBV if you have any good contacts there.
Yes, I'll do. In my current project, I think I'll write my own MARC-client and then transcode the records before importing them. We mainly have german books, but also a few russian, chinese, japanese and greek. The current database is a flatfile where foreign books aren't included at all. Again, can anybody name me a reliable UTF-8 data source? I just want to test wether my UTF setup is working throughout the system. Dorian Meid
Hello, This thread stopped more than 6 months ago, yet I do not know what happened next. I am very interested to a workable solution to this problem: half of our books are in French and we have exactly the same problem: z39.50 records fetched from (bilingual) canadian libraries do not get properly in our ISO koha. All accented letters are wrongly interpreted when they show up in our MARC21 records. I am not even sure the 3 Canadian libraries use UTF8 although it looks like it. A reliable source of UTF8 encoding would clear the question. Michel Virard Montreal Dorian Meid wrote:
Thanks for validating this!
Am 13.03.2007 um 12:14 schrieb Thomas Dukleth: ...
I have tested and found that the problem is with GBV at least for this record. I had never tested their records retuned in UTF-8 from port 20012 previously. ... Please report this problem to GBV if you have any good contacts there.
Yes, I'll do.
In my current project, I think I'll write my own MARC-client and then transcode the records before importing them. We mainly have german books, but also a few russian, chinese, japanese and greek. The current database is a flatfile where foreign books aren't included at all.
Again, can anybody name me a reliable UTF-8 data source? I just want to test wether my UTF setup is working throughout the system.
Dorian Meid
_______________________________________________ Koha-devel mailing list Koha-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/koha-devel
-- View this message in context: http://www.nabble.com/UTF-8-tf3382955.html#a14122458 Sent from the Koha - Dev mailing list archive at Nabble.com.
participants (3)
-
Dorian Meid -
Michel Virard -
Thomas Dukleth