Converting Koha sources to UTF-8
I would like to convert the Koha source tree to UTF-8. Having everything in a systematic, modern encoding would be beneficial for everyone, I think. I wrote a script to find files that are not in UTF-8 (attached for review); it uses the isutf8 tool from Joey Hess's moreutils package (see http://kitenet.net/~joey/code/moreutils/). The script excludes a number of files based on the suffix, to avoid confusing things by reporting binary files, etc. It currently reports 59 files for me. Most are just copyright symbols or names in release notes, and all of those seem to be in the ISO-8859-1 (Latin-1) character set, so converting them is easy. I did the actual conversion with the "iconv -f ISO-8859-1 -t UTF-8" command. The following three files puzzle me, however: C4/tests/testrecords/marc21_marc8_combining_chars.dat etc/zebradb/etc/urx.chr etc/zebradb/lang_defs/en/sort-string-utf.chr Is it acceptable to conver them to UTF-8, or should they remain as they are? I don't know how they are used.
Hi, 2010/3/24 Lars Wirzenius <lars@catalyst.net.nz>:
C4/tests/testrecords/marc21_marc8_combining_chars.dat
This should be left alone - it is a test file for converting MARC records from the MARC-8 character set to UTF-8.
etc/zebradb/etc/urx.chr etc/zebradb/lang_defs/en/sort-string-utf.chr
This can be converted to UTF-8, but not mechanically. They're Zebra configurations that are expected to be in ISO-8859-1 by default. To signal that they're using UTF-8, the following directive would have to be added to them and the results of the indexing tested: encoding utf-8 Regards, Galen -- Galen Charlton gmcharlt@gmail.com
In my haste, I already sent out a series of patches that take care of the easy cases of UTF-8 conversion. On Fri, 2010-03-26 at 07:45 -0700, Galen Charlton wrote:
Hi,
2010/3/24 Lars Wirzenius <lars@catalyst.net.nz>:
C4/tests/testrecords/marc21_marc8_combining_chars.dat
This should be left alone - it is a test file for converting MARC records from the MARC-8 character set to UTF-8.
Right. My patches include a find-nonutf8 script, which ignores those files now.
etc/zebradb/etc/urx.chr etc/zebradb/lang_defs/en/sort-string-utf.chr
This can be converted to UTF-8, but not mechanically. They're Zebra configurations that are expected to be in ISO-8859-1 by default. To signal that they're using UTF-8, the following directive would have to be added to them and the results of the indexing tested:
encoding utf-8
Since I am still getting up to speed with Koha development, I am not sure I can test this reliably yet. My patches also ignore these files, which I think will be fine for now.
participants (2)
-
Galen Charlton -
Lars Wirzenius