Hi folks, I've been looking at merging Biblio.pm from dev-week and HEAD and it looks like there are a few choices that need to be made ... I'd like to get your feedback on these. (BTW: Tumer, go ahead and commit all the Biblio.pm stuff you've got to HEAD, we'll deal with the conflicts after the fact ...) 1. in HEAD we have 'z3859_extended_services', a routine capable of handling all the Z3859 extended services offered by Zebra, as well as 'set_service_options', which sets the service options before calling z3859_extended_services. But in dev_week, it looks like that's handled with 'zebraop' and when zebraop fails it falls back on zebraopfiles ... So which of these routes shall we take, perhaps a combo of both approaches? 2. in HEAD we have no more bibid, is that going to cause any strange conflicts in dev_week code (didn't spot anything specific). 3. in HEAD we have 3 types of subs, REALXXX, NEWXXX, something_elseXXX ... can't we just have one type of sub? Any reason we can't just completely clean up this API once and for all? 4. char_decode (you knew it was coming :-). Tumer, can you check the mail I sent earlier about encoding and see if installing XML::SAX::Expat solves your problems with MARC::File::XML's handling of encoding? If we have to stick with char_decode, then so be it, but I'd really like to have a more standards-compliant way to handle encoding in both MARC21 and UNIMARC. What this means is that in addition to managing the proper encodings (to and from MARC-8 in MARC21, and to and from the appropriate encodings in UNIMARC), Koha should also handle the leader (for MARC21) and tag 100 (for UNIMARC), and properly set the encoding when all is said and done. Internally, I vote that we store everything as UTF-8 in all circumstances. That's all for now .. Cheers, -- Joshua Ferraro VENDOR SERVICES FOR OPEN-SOURCE SOFTWARE President, Technology migration, training, maintenance, support LibLime Featuring Koha Open-Source ILS jmf@liblime.com |Full Demos at http://liblime.com/koha |1(888)KohaILS
Hi Josh, Regarding difference of dev-week biblio.pm and HEAD biblio.pm 1- Dev-week biblio.pm has got $bibid variable but only as a variable name. There are no marc_biblio, marc_word or marc_subfield tables that they use. All the version 2_2 function names are kept to allow 2_2 intranet & opac scripts to work without modifications. So going ahead and doing a search&replace all $bibid to $biblionumber will not break anything. 2- Dev-week biblio.pm contains lots of the unnecessary code left over from 2_2 3- Dev-week biblio contains a routine called zebraop which only gets called twice.Once for 'specialUpdate' updating the zebradatabase and once for 'recordDelete' deleting the database. All handling of by which user to which port is done at Context.pm level redaing the configuration file koha.conf or newly renamed to koha.xml (by me) to prevent confusion with the 2_2 koha.conf. New one is an xml file which contains both mysql and zebra configurations. 4- Dev-week biblio.pm also contains some new code to be able to update holdings data in marc records (such as onloan and holdingbranch). 5-Dev-week biblio also contains sorting of callnumbers according to LC rules by default (NEU uses LC) which will have to be looked into and set rules on how to sort for Dewey,LC or any other. Another discussion subject 6-HEAD biblio.pm is clean of unnecessary code. All but a few $bibid variables are removed 7-HEAD biblio.pm introduced new routines strating with REAL thus changed the logic in database handling. I could not get head to work properly so I do not know whether new bugs introduced or not. 8-Adding zebra support to any biblio.pm is very easy with zebraop. Any time you write a marc record to the database just call zebraop with the biblionumber. Any time you want to delete a marc record from the database call zebraop prior to actual deletion. 9-As long as HEAD biblio.pm code works with mysql database properly just adding zebrasupport for it should not be difficult. 10-I also we should clean up this biblio once and for all. But if you need zebrasupport today by simlply adding a few fields to your existing koha db then dev-week biblio.pm should be kept otherwise for a clean 3.0 HEAD biblio.pm forge dev-week biblio and start debugging and redesigning HEAD biblio.pm. I personally will not be able to do that as I have production on dev-week biblio.pm and now adding zebra support to AuthoritiesMarc.pm which by the way uses zebraop logic 11-Regarding char_decode: I already have Expat installed and still could not get M:F:X to work properly. But I also think that we should get rid of it otherwise its always going to be a problem as new formats or languages are added to KOHA. I also stroongly believe that all data within KOHA & ZEBRA should be UTF-8 and non UTF-8 handling should only be allowed when importing or exporting of MARC records Cheers Tumer -----Original Message----- From: koha-devel-bounces+tgarip=neu.edu.tr@nongnu.org [mailto:koha-devel-bounces+tgarip=neu.edu.tr@nongnu.org] On Behalf Of Joshua Ferraro Sent: Wednesday, May 24, 2006 3:38 PM To: koha-devel@nongnu.org Subject: [Koha-devel] Biblio.pm Choices IMPORTANT Hi folks, I've been looking at merging Biblio.pm from dev-week and HEAD and it looks like there are a few choices that need to be made ... I'd like to get your feedback on these. (BTW: Tumer, go ahead and commit all the Biblio.pm stuff you've got to HEAD, we'll deal with the conflicts after the fact ...) 1. in HEAD we have 'z3859_extended_services', a routine capable of handling all the Z3859 extended services offered by Zebra, as well as 'set_service_options', which sets the service options before calling z3859_extended_services. But in dev_week, it looks like that's handled with 'zebraop' and when zebraop fails it falls back on zebraopfiles ... So which of these routes shall we take, perhaps a combo of both approaches? 2. in HEAD we have no more bibid, is that going to cause any strange conflicts in dev_week code (didn't spot anything specific). 3. in HEAD we have 3 types of subs, REALXXX, NEWXXX, something_elseXXX ... can't we just have one type of sub? Any reason we can't just completely clean up this API once and for all? 4. char_decode (you knew it was coming :-). Tumer, can you check the mail I sent earlier about encoding and see if installing XML::SAX::Expat solves your problems with MARC::File::XML's handling of encoding? If we have to stick with char_decode, then so be it, but I'd really like to have a more standards-compliant way to handle encoding in both MARC21 and UNIMARC. What this means is that in addition to managing the proper encodings (to and from MARC-8 in MARC21, and to and from the appropriate encodings in UNIMARC), Koha should also handle the leader (for MARC21) and tag 100 (for UNIMARC), and properly set the encoding when all is said and done. Internally, I vote that we store everything as UTF-8 in all circumstances. That's all for now .. Cheers, -- Joshua Ferraro VENDOR SERVICES FOR OPEN-SOURCE SOFTWARE President, Technology migration, training, maintenance, support LibLime Featuring Koha Open-Source ILS jmf@liblime.com |Full Demos at http://liblime.com/koha |1(888)KohaILS _______________________________________________ Koha-devel mailing list Koha-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/koha-devel
Hi Tumer, Thanks for your speedy response ... On Wed, May 24, 2006 at 06:37:12PM +0300, Tümer Garip wrote:
1-10 Thanks!
11-Regarding char_decode: I already have Expat installed and still could not get M:F:X to work properly. But I also think that we should get rid of it otherwise its always going to be a problem as new formats or languages are added to KOHA. I also stroongly believe that all data within KOHA & ZEBRA should be UTF-8 and non UTF-8 handling should only be allowed when importing or exporting of MARC records I've tried running some of your records through a 'roundtrip' script that tests both as_xml and new_from_xml and their encoding/re-encoding abilities ... and they seem to work fine for me.
Here's the script: http://liblime.com/public/roundtrip.pl and here are the records I tested: http://liblime.com/public/turkish.iso2709 Run it like this: $./roundtrip.pl turkish.iso2709 turkish.utf8.iso2709 error.xml error.mrc If it throws an error it will dump out the offending record in MARCXML and in MARC format. Let me know how it works for you. I completely agree that all data within Koha 3.0 should be stored as UTF-8 and that everything that's not UTF-8 should be handled at import or export. However, currently, the only module that handles MARC-8 encoding properly is MARC::Charset. Since 90% of all MARC21 records are in MARC-8, it's very important that we handle it properly. Also, since MARC-8 and UTF-8 are the only valid encodings for MARC21, and since the MARC::Record suite already handles records in these encodings correctly, I think we should use it. Up till now, Koha has not been managing MARC-8 encoded records at all! Now, as to what to do for non-MARC21 records, I think the solution is to expand the capabilities of the MARC::Record suite to include support for UNIMARC encodings, etc. Some work has already been done on this ... Now, there's another matter of what to do with other encodings within MARC21 ... because obviously there are records out there in MARC21 format that have been incorrectly encoded. In my opinion, we should expand the MARC::Record suite to include encoding management in a 'from -> to' fashion, so that it's possible to re-encode incorrectly encoded records as UTF-8. The key in all of this IMO is to keep encoding management out of the ILS. Isn't that the whole point of using external toolkits? Let me know what you think. Cheers, -- Joshua Ferraro VENDOR SERVICES FOR OPEN-SOURCE SOFTWARE President, Technology migration, training, maintenance, support LibLime Featuring Koha Open-Source ILS jmf@liblime.com |Full Demos at http://liblime.com/koha |1(888)KohaILS
Cheers Tumer
-----Original Message----- From: koha-devel-bounces+tgarip=neu.edu.tr@nongnu.org [mailto:koha-devel-bounces+tgarip=neu.edu.tr@nongnu.org] On Behalf Of Joshua Ferraro Sent: Wednesday, May 24, 2006 3:38 PM To: koha-devel@nongnu.org Subject: [Koha-devel] Biblio.pm Choices IMPORTANT
Hi folks,
I've been looking at merging Biblio.pm from dev-week and HEAD and it looks like there are a few choices that need to be made ... I'd like to get your feedback on these.
(BTW: Tumer, go ahead and commit all the Biblio.pm stuff you've got to HEAD, we'll deal with the conflicts after the fact ...)
1. in HEAD we have 'z3859_extended_services', a routine capable of handling all the Z3859 extended services offered by Zebra, as well as 'set_service_options', which sets the service options before calling z3859_extended_services. But in dev_week, it looks like that's handled with 'zebraop' and when zebraop fails it falls back on zebraopfiles ...
So which of these routes shall we take, perhaps a combo of both approaches?
2. in HEAD we have no more bibid, is that going to cause any strange conflicts in dev_week code (didn't spot anything specific).
3. in HEAD we have 3 types of subs, REALXXX, NEWXXX, something_elseXXX ... can't we just have one type of sub? Any reason we can't just completely clean up this API once and for all?
4. char_decode (you knew it was coming :-). Tumer, can you check the mail I sent earlier about encoding and see if installing XML::SAX::Expat solves your problems with MARC::File::XML's handling of encoding? If we have to stick with char_decode, then so be it, but I'd really like to have a more standards-compliant way to handle encoding in both MARC21 and UNIMARC. What this means is that in addition to managing the proper encodings (to and from MARC-8 in MARC21, and to and from the appropriate encodings in UNIMARC), Koha should also handle the leader (for MARC21) and tag 100 (for UNIMARC), and properly set the encoding when all is said and done. Internally, I vote that we store everything as UTF-8 in all circumstances.
That's all for now ..
Cheers,
-- Joshua Ferraro VENDOR SERVICES FOR OPEN-SOURCE SOFTWARE President, Technology migration, training, maintenance, support LibLime Featuring Koha Open-Source ILS jmf@liblime.com |Full Demos at http://liblime.com/koha |1(888)KohaILS
Joshua Ferraro <jmf@liblime.com> wrote:
1. in HEAD we have 'z3859_extended_services', a routine capable of handling all the Z3859 extended services offered by Zebra, as well as 'set_service_options', which sets the service options before calling z3859_extended_services. But in dev_week, it looks like that's handled with 'zebraop' and when zebraop fails it falls back on zebraopfiles ...
What is zebraop?
So which of these routes shall we take, perhaps a combo of both approaches?
[...]
Internally, I vote that we store everything as UTF-8 in all circumstances.
I agree. It costs us maybe some CPU and maybe some space, but it probably will stops us all going mad. The only sensible alternative is to have general string accessor functions, but I'd expect that to use more CPU in reality. Hope that helps, -- MJ Ray - personal email, see http://mjr.towers.org.uk/email.html Work: http://www.ttllp.co.uk/ irc.oftc.net/slef Jabber/SIP ask
participants (3)
-
Joshua Ferraro -
MJ Ray -
Tümer Garip