- Koha-devel - lists.koha-community.org

Encoding Problems and Solutions (IMPORTANT!)
by Joshua Ferraro 20 May '06

20 May '06

Hi all, I've spent the better part of two days working on the encoding problems that several of the Koha developers have been experiencing and I think I've met with some success. First, software I'm running: Perl 5.8.4 XML::SAX 0.14 MARC::Record 2.0RC1 MARC::File::XML 0.83 MARC::Charset 0.95 There were two problems ... the first happens when you have MARC21 records with wrongly encoded characters. Let me be clear that I mean wrongly encoded by the MARC21 MARC-8 encoding. The LOC has defined all of the valid MARC-8 characters and mapped them to UTF-8 here: http://www.loc.gov/marc/specifications/codetables.xml (warning, very large file) This is the code table that MARC::Charset uses to convert MARC-8 encoded records into UTF-8 ... and that's what MARC::File::XML uses to convert a binary MARC file into MARCXML. By default, if MARC::Charset encounters a character that isn't in that code table, it drops the whole subfield and throws a warn like this: no mapping found at position 8 in Price : � 7.99; Inv.# B 476913; Date 06/03/98; Supplier : Dawson UK; Recd 20/03/98; Contents : 1. The problem : 1. Don't bargain over positions; 2. The method : 2. Separate the people from the problem; 3. Focus on interests, not positions; 4. Invent options for mutual gain; 5. Insist on using objective criteria; 3. Yes, but : 6. What if they are more powerful? 7. What if they won't play? 8. What if they use dirty tricks? 4. In conclusion; 5. Ten questions people ask about getting to yes; g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/local/share/perl/5.8.4/MARC/Charset.pm line 197. I don't know if you can see it or not, but before the 7.99 in the above dump is a \x9C character, which is an invalid MARC-8 character. The temporary solution is to add the following: use MARC::Charset; MARC::Charset->ignore_errors(1); to any script or module that you expect to encounter wrong encodings. This way, it will just drop the offending character rather than the whole subfield (it still throws the warn though). I'm not 100% happy with that solution, but I can deal with it until someone has better suggestion. The second problem is the more serious one ... several of us have tried to pass UTF-8 encoded XML records in to the new_from_xml() method and had the parser crash if there were 'combining characters' in the record. The problem seems to be with the PurePerl parser ... as soon as I installed XML::SAX::Expat it went away. I've attached a script to this email that you can use to test your system to make sure things are set up correctly. The script attempts to convert a binary MARC record in either UTF-8 or MARC-8 encoding, to MARC-XML (encoded as UTF-8) and then back to binary MARC (as UTF-8). You can test the first problem with the following record: http://liblime.com/public/badencoding.mrc And the second with this one: http://liblime.com/public/combiningchar.mrc Run the script like this: $ ./roundtrip.pl badencoding.mrc out.utf8.mrc dump Compare the original MARC record with the out.utf8.mrc ... the only difference should be in the encoding and possibly missing chars if your records had bad encoding. (you can edit roundtrip.pl to turn on/off the ignore_errors flag). And don't forget to install XML::SAX::Expat. So ... now that we've got the MARC21 encoding problems out of the way, we need to look at UNIMARC. Mike Rylander has already done some work on this but I think due to the second error above, we've not had success testing thusfar. Cheers, -- Joshua Ferraro VENDOR SERVICES FOR OPEN-SOURCE SOFTWARE President, Technology migration, training, maintenance, support LibLime Featuring Koha Open-Source ILS jmf(a)liblime.com |Full Demos at http://liblime.com/koha |1(888)KohaILS

1 0

NEW branch on rel_2_2 : authorities
by Henri-Damien LAURENT 16 May '06

16 May '06

Hi, I am working for a network which needs a pretty high standard stuff on authorities with multi hierarchical authorities and a good summary feature. We were planning to do it on head but I have a customer review on June, 29. and we fear authorities management in HEAD won't be fair enough by that time. But we still want Koha stable branches to remain stable. All the work I shall do will be on a new branch called hdl_auth. This will allow me to create two subs in Authorities called build_summary and build_hierarchy. Both will be called whenever an authority is changed and shall store datas into auth_header tables new fields. It seems to me that build a summary on the fly would be time consuming (and tümer seemed to agree to that principle) and building a hierarchy is a problem that requires many authorities_search for 1 item. So I will create a branch. I will propose you to test when development process is over. I shall have to get it work into version 3.0. Bonne journée. -- Henri-Damien LAURENT

2 1

smart bot on IRC
by Pierrick LE GALL 16 May '06

16 May '06

Hello koha-devel, This message is for chris but I'm sure it will be interesting for every #koha channel user... Chris, just as a reminder, can you install a "smart" bot on #koha. So that we could ask "smartbot, what about notforloan?" and the bot would tell all what he as registered concerning the loan prevention. Bye -- Pierrick LE GALL INEO media system

2 1

IBERMARC addition
by Damián Marqués Lluch 15 May '06

15 May '06

Last April, 24th I tried to join to this list and I posted a message. However, somehow, I got added to the sourceforge devel-list (apparently no longer in use) and probably my message was never read. I reproduce my message below. On the other hand, I'm not subscribed to the right mailing list, so I will not receive any reply... does anybody know how can I subscribe to the right mailing list? Dear all, I recently downloaded koha-2.2.5 from your website and started to play around with it. I'm not a librarian, I'm just a computer science guy who was looking a program to keep track of his personal library, photography and music CD collection. All in a single program. MARC format seems to be suitable. Since I'm spanish, I took the MARC21 structure as a basis and changed it to the IBERMARC format, the one in use by the Biblioteca Nacional de España (Spanish National Library). Furthermore, I also took the IBERMARC format for authority records and I feed it into the proper MySQL koha tables (auth*). Finally I made an spanish stopwords list. So, I've changed the koha-2.2.5.tar.gz file in my computer in the following way: added: koha-2.2.5/scripts/misc/marc_datas/ibermarc_es/structure_def.sql added: koha-2.2.5/scripts/misc/sql-datas/spanish/authorities_normal_ibermarc.sql added: koha-2.2.5/scripts/misc/sql-datas/spanish/authorities_normal_ibermarc.txt added: koha-2.2.5/scripts/misc/sql-datas/spanish/stopwords.sql added: koha-2.2.5/scripts/misc/sql-datas/spanish/stopwords.txt changed: koha-2.2.5/Install.pm The changes made to Install.pm allow the user to select option 3. IBERMARC so that the default framework is IBERMARC. All new files have been encoded as ISO-8859-1 in order for the spanish character set to be fully recognized (in particular Unicode UTF-8 was not working correctly in my system). Install.pm has been left Unicode UTF-8 All of this is working in my system, so I wonder if you could be interested in have a look to my changes. Maybe you could consider the posibility to add them to the official distribution after testing them. With kind regards, Damián

3 5

notforloan rule
by Pierrick LE GALL 15 May '06

15 May '06

Hi koha-devel, Working on C4::Reserves2 and associated scripts, I'm wondering what is the rule to say an item is not for loan? In the database, we have itemtypes.notforloan and items.notforloan. In the authorised values for category "STATUS" I have 5 values from 0 to 5 (with no 3). In my items table, most of my items have NULL as notforloan value. In my itemtypes table, notforloan is either "loanable" or "excluded from loan". Should I consider that if items.notforloan is null then the correct value is the itemtypes one? Bye -- Pierrick LE GALL INEO media system

2 2

Restructuring the Wiki to make use of Namespaces
by Russel Garlick 10 May '06

10 May '06

Hi everyone, Just a note to warn everyone who is subscribed to the RSS feed of the Wiki that we will be playing around with it some more today. Pierrick, Tina and I will be implementing name spaces to help structure the wiki a little bit more. http://wiki.koha.org/doku.php?id=wiki:syntax#internal http://wiki.splitbrain.org/wiki%3Anamespaces So you can expect a bit more traffic on that feed. Cheers Russel

1 0

devWeek : question
by Paul POULAIN 27 Apr '06

27 Apr '06

Hello all, for devWeek we will have a wifi access AND a non wifi access. To enable dhcp (that is secured), could everybody send me their MAC addess by private mail. thanks a lot. -- Paul POULAIN et Henri Damien LAURENT Consultants indépendants en logiciels libres et bibliothéconomie (http://www.koha-fr.org)

1 0

Re: Reserve System
by paul.poulain＠free.fr 26 Apr '06

26 Apr '06

(mail by webmail) >>Pierrick & al, pls note that reserve system has been *deeply modified* >> by SAN-OP. The code should be in CVS soon [...] > Will you explain what is modified and why before commiting? Yes. I already wrote something on March, 29th, "new feature to come soon". I agree that "many improvements in transfer management" has to be explained a little bit more ;-) -- Paul

1 0

Reserve system
by Pierrick LE GALL 26 Apr '06

26 Apr '06

Hi koha-devel, During my investigation on bug 668 [1], I've started discovering how reserve worked. I must admit I'm a bit disappointed about several points: 1. A reserve can be linked to a biblio or to an item. I've asked hdl on IRC in which case it was useful to reserve a specific item, he told me that was to avoid preventing an issue on another branch. My opinion is that this is not the right design if branches independance is the reason of item number in reserve table. 2. Reserve cancellation can only be made when reserve is linked to an item (this is the origin of the problem, coming from the correction on the bug 969 [2]). When trying to issue an item reserved, only the first reserve is displayed for confirmation of cancel and only the first reserve is cancelled if asked by the librarian, without changing the priority of other reserves. I think it's a bug of course. 3. Finally, I really don't understand why Koha ask for reserve cancellation when issuing. A warning isn't enough? Why would a librarian cancel the reserve of borrower B1 if she authorizes B2 to issue the item? It doesn't make sense to me. Please help me to understand the deep origin of this design I don't understand :-) [1] http://bugs.koha.org/cgi-bin/bugzilla/show_bug.cgi?id=668 [2] http://bugs.koha.org/cgi-bin/bugzilla/show_bug.cgi?id=969 Bye -- Pierrick LE GALL INEO media system

3 5

Re: Reserve System
by paul.poulain＠free.fr 26 Apr '06

26 Apr '06

(reading my mails from webmail) Hi all,Koha-devel(a)nongnu.org Pierrick & al, pls note that reserve system has been deeply modified by SAN-OP. The code should be in CVS soon, but the developper is in holiday & I can't understand some things he sended to me before leaving. I still hope it will be done just before devWeek -- Paul

2 1