Hi Tumer, Thanks for your speedy response ... On Wed, May 24, 2006 at 06:37:12PM +0300, Tümer Garip wrote:
1-10 Thanks!
11-Regarding char_decode: I already have Expat installed and still could not get M:F:X to work properly. But I also think that we should get rid of it otherwise its always going to be a problem as new formats or languages are added to KOHA. I also stroongly believe that all data within KOHA & ZEBRA should be UTF-8 and non UTF-8 handling should only be allowed when importing or exporting of MARC records I've tried running some of your records through a 'roundtrip' script that tests both as_xml and new_from_xml and their encoding/re-encoding abilities ... and they seem to work fine for me.
Here's the script: http://liblime.com/public/roundtrip.pl and here are the records I tested: http://liblime.com/public/turkish.iso2709 Run it like this: $./roundtrip.pl turkish.iso2709 turkish.utf8.iso2709 error.xml error.mrc If it throws an error it will dump out the offending record in MARCXML and in MARC format. Let me know how it works for you. I completely agree that all data within Koha 3.0 should be stored as UTF-8 and that everything that's not UTF-8 should be handled at import or export. However, currently, the only module that handles MARC-8 encoding properly is MARC::Charset. Since 90% of all MARC21 records are in MARC-8, it's very important that we handle it properly. Also, since MARC-8 and UTF-8 are the only valid encodings for MARC21, and since the MARC::Record suite already handles records in these encodings correctly, I think we should use it. Up till now, Koha has not been managing MARC-8 encoded records at all! Now, as to what to do for non-MARC21 records, I think the solution is to expand the capabilities of the MARC::Record suite to include support for UNIMARC encodings, etc. Some work has already been done on this ... Now, there's another matter of what to do with other encodings within MARC21 ... because obviously there are records out there in MARC21 format that have been incorrectly encoded. In my opinion, we should expand the MARC::Record suite to include encoding management in a 'from -> to' fashion, so that it's possible to re-encode incorrectly encoded records as UTF-8. The key in all of this IMO is to keep encoding management out of the ILS. Isn't that the whole point of using external toolkits? Let me know what you think. Cheers, -- Joshua Ferraro VENDOR SERVICES FOR OPEN-SOURCE SOFTWARE President, Technology migration, training, maintenance, support LibLime Featuring Koha Open-Source ILS jmf@liblime.com |Full Demos at http://liblime.com/koha |1(888)KohaILS
Cheers Tumer
-----Original Message----- From: koha-devel-bounces+tgarip=neu.edu.tr@nongnu.org [mailto:koha-devel-bounces+tgarip=neu.edu.tr@nongnu.org] On Behalf Of Joshua Ferraro Sent: Wednesday, May 24, 2006 3:38 PM To: koha-devel@nongnu.org Subject: [Koha-devel] Biblio.pm Choices IMPORTANT
Hi folks,
I've been looking at merging Biblio.pm from dev-week and HEAD and it looks like there are a few choices that need to be made ... I'd like to get your feedback on these.
(BTW: Tumer, go ahead and commit all the Biblio.pm stuff you've got to HEAD, we'll deal with the conflicts after the fact ...)
1. in HEAD we have 'z3859_extended_services', a routine capable of handling all the Z3859 extended services offered by Zebra, as well as 'set_service_options', which sets the service options before calling z3859_extended_services. But in dev_week, it looks like that's handled with 'zebraop' and when zebraop fails it falls back on zebraopfiles ...
So which of these routes shall we take, perhaps a combo of both approaches?
2. in HEAD we have no more bibid, is that going to cause any strange conflicts in dev_week code (didn't spot anything specific).
3. in HEAD we have 3 types of subs, REALXXX, NEWXXX, something_elseXXX ... can't we just have one type of sub? Any reason we can't just completely clean up this API once and for all?
4. char_decode (you knew it was coming :-). Tumer, can you check the mail I sent earlier about encoding and see if installing XML::SAX::Expat solves your problems with MARC::File::XML's handling of encoding? If we have to stick with char_decode, then so be it, but I'd really like to have a more standards-compliant way to handle encoding in both MARC21 and UNIMARC. What this means is that in addition to managing the proper encodings (to and from MARC-8 in MARC21, and to and from the appropriate encodings in UNIMARC), Koha should also handle the leader (for MARC21) and tag 100 (for UNIMARC), and properly set the encoding when all is said and done. Internally, I vote that we store everything as UTF-8 in all circumstances.
That's all for now ..
Cheers,
-- Joshua Ferraro VENDOR SERVICES FOR OPEN-SOURCE SOFTWARE President, Technology migration, training, maintenance, support LibLime Featuring Koha Open-Source ILS jmf@liblime.com |Full Demos at http://liblime.com/koha |1(888)KohaILS