March 2006 - Koha-devel - lists.koha-community.org

Re: Unimarc, marc21, Unicode, and MARC::File::XML
by Paul POULAIN 21 Mar '06

21 Mar '06

Mike Rylander a écrit : > I tested with the record you sent Ed and me, and everything seems to > work for me ... > As you can see, I tested several variants of the UNIMARC flag, and > even tested not sending the encoding to new_from_xml() ... it all > seems to work for me, and I'm not sure what problems you're seeing. > Perhaps you just needed to set your binmode for the XML source? strange, strange... What does my script : * retrieve the MARC::Record from zebra * read some datas from mysql * build a page with HTML::Template * send the pages to the browser I added 3 lines to save the record in a file after reading from zebra. Adding binmode(F,':utf8'); before saving my record in F, give me correct UTF-8. without binmode, it's NOK. But when I put the MARC::record in a page builded with HTML::Template, it's wrong. The HTML is utf-8 (html page encoding). It also contains some strings from mySQL and all strings from mySQL appear as correct utf8 while all strings coming from the MARC::record coming from zebra are not ! I can add "binmode()" to the template output, but everything goes wrong with strings from mySQL. Any suggestion welcomed ! -- Paul POULAIN et Henri Damien LAURENT Consultants indépendants en logiciels libres et bibliothéconomie (http://www.koha-fr.org)

3 5

Updates to Updating Koha document
by Stephen Hedges 17 Mar '06

17 Mar '06

Pierrick Le Gall has made some significant revisions to Joshua Ferraro's "Updating Koha" document. Find it on www.kohadocs.org under System Administration -> Maintenance and Troubleshooting. -- Stephen Hedges Skemotah Solutions, USA www.skemotah.com -- shedges(a)skemotah.com

2 2

Building zebradb
by Tümer Garip 17 Mar '06

17 Mar '06

Hi, We have now put the zebra into production level systems. So here is some experience to share. Building the zebra database from single records is a veeeeery looong process. (100K records 150k items) Best method we found: 1- Change zebra.cfg file to include iso2079.recordType:grs.marcxml.collection recordType:grs.xml.collection 2- Write (or hack export.pl) to export all the marc records as one big chunk to the correct directory with an extension .iso2079 And system call "zebraidx -g iso2079 -d <dbnamehere> update records -n". This ensures that zebra knows its reading marc records rather than xml and builds 100K+ records in zooming speed. Your zoom module always uses the grs.xml filter while you can anytime update or reindex any big chunk of the database as long as you have marc records. 3-We are still using the old API so we read the xml and use MARC::Record->new_from_xml( $xmldata ) A note here that we did not had to upgrade MARC::Record or MARC::Charset at all. Any marc created within KOHA is UTF8 and any marc imported into KOHA (old marc_subfield_tables) was correctly decoded to utf8 with char_decode of biblio. 4- We modified circ2.pm and items table to have item onloan field and mapped it to marc holdings data. Now our opac search do not call mysql but for the branchname. 5- Average updates per day is about 2000 (circulation+cataloger). I can say that the speed of the zoom search which slows down during a commit operation is acceptable considering the speed gain we have on the search. 6- Zebra behaves very well with searches but is very tempremental with updates. A queue of updates sometimes crashes the zebraserver. When the database crash we can not save anything even though we are using shadow files. I'll be reporting on this issue once we can isolate the problems. Regards, Tumer

5 8

Re: MARC Label in English for UNIMARC
by paul POULAIN 16 Mar '06

16 Mar '06

roberto.sturman(a)ts.infn.it a écrit : >>From: Alan Hopkinson >>Subject: [Koha-devel] RE: MARC Label in English >>Date: Fri, 03 Mar 2006 10:09:53 +0000 >> >>I hope that a version of KOHA will be done for UNIMARC as well as MARC21. >> >> >It >is widely used in Europe, the Far East and developing countries >particularly > > >>outside the Anglo-American world. >> >>Alan Hopkinson >>Chair, IFLA Permanent UNIMARC Committee >> >> >Here is my contribution to the KOHA project. > > (attached where complete UNIMARC in english files) >Some weeks ago I prepared the UNIMARC definition files for KOHA, both for >Bibliographic and Authorities records. > > Many thanks for everybody that could be interested by unimarc in english. I cc koha-devel & koha mailing lists. it seems you sended only bibliographic files. To export authorities files, look in auth_*_structure tables. could you please dump everything again (in separate files), with -c to have field names : this avoid having unnamed insert : if the DB changes, the file could'nt be imported anymore. Once i have them, i'll include everything in CVS for next stable version, and ask ppl on mailing list for some tests. Note : It seems you have only used a few from existing UNIMARC plugins. We have one for every coded fields (1xx), although it's only in french for instance. If you candidate to translate those, I could send the templates to you as well. -- Paul POULAIN et Henri Damien LAURENT Consultants indépendants en logiciels libres et bibliothéconomie (http://www.koha-fr.org)

1 0

Daily fines - Koha 2.2.4 W32
by Siop Gopïo | Copy Shop 15 Mar '06

15 Mar '06

With apologies for cross-posting. I would like to know if anyone using Koha has managed to set up fines that are applied each night, regardless of how many days the item is overdue by. Put simply, what I'm looking for is: total fine = daily fine amount (multiplied by) days overdue I don't want the fine to stop being applied after 21 days or whatever, I'd just like it to continue. Does anyone have such a setup? Could you send me any related scripts/files? I'm using Koha 2.2.4 W32 R0, if it makes a difference. I'm not a coder myself (although I'm not scared of having a go at editing files), but just can't get my head around the default fines scripts that come with Koha. Regards, Chris Hearn Rheolwr y Siop Gopïo Copy Shop Manager ----------------------------------------------- Siop Gopïo Urdd Myfyrwyr Aberystwyth mewn partneriaeth â Call Print http://www.urddaber.co.uk/gwasanaethau/siopgopi E-bost: undeb.siopgopi(a)aber.ac.uk Yr Undeb, Penglais, Aberystwyth, SY23 3DX Ffôn: 01970 621705 Ffacs: 01970 621724 Aberystwyth Guild of Students Copy Shop in partnership with Call Print www.aberguild.co.uk/services/copyshop E-mail: union.copyshop(a)aber.ac.uk The Union, Penglais, Aberystwyth, SY23 3DX Phone: 01970 621705 Fax: 01970 621724

1 0

Koha Meeting Minutes
by Joshua Ferraro 13 Mar '06

13 Mar '06

Hi all, Here are the meeting minutes from today's meeting. There were a few questions for Paul, so Paul, if you would comment that'd be great. present: thd, pierrick, russ, chris, shedges, oleonard, kados NEWS & QUESTIONS Pierrick asked how we currently manage new features. We discussed the fairly informal process as it stands. Pierrick recommended that new features be speced out in the wiki and announced on koha-devel so that other developers can comment on them. We discussed the need for more QA testing when developing new features. Questions for Paul: when can we begin announcing the Koha Con? Are the dates and venues confirmed? can we get details? :-) PERL-ZOOM INTEGRATION Chris worked more on searching last week, excepting the 'MARC Search', the perl-zoom plugin supports all the same searches. deletions still haven't been finished, and the code needs testing / tweaking UTF-8 our tentitive conclusion is that paul's troubles with UTF-8 were due to non-UTF-8 data in his database. We will investigate further and confirm this to make sure. VERSION CONTROL SYSTEM Discussed briefly the limitations of CVS and the desire to move to SVN or GNU ARCH. Savannah is hard at work getting SVN to work and when it looks like that stabilzes we'll take a look at it. if I missed anything let me know. Cheers, -- Joshua Ferraro VENDOR SERVICES FOR OPEN-SOURCE SOFTWARE President, Technology migration, training, maintenance, support LibLime Featuring Koha Open-Source ILS jmf(a)liblime.com |Full Demos at http://liblime.com/koha |1(888)KohaILS

1 0

RE: MARC Label in English
by Alan Hopkinson 13 Mar '06

13 Mar '06

I hope that a version of KOHA will be done for UNIMARC as well as MARC21. It is widely used in Europe, the Far East and developing countries particularly outside the Anglo-American world. Alan Hopkinson Chair, IFLA Permanent UNIMARC Committee > -----Original Message----- > From: maolins(a)gmail.com [mailto:maolins@gmail.com] > Sent: Friday, March 03, 2006 9:46 AM > To: koha-devel(a)nongnu.org > Cc: Alan Hopkinson > Subject: Re: MARC Label in English > > > Hi all and Mr. Hopkinson from PUC, > > > 2006/3/3, Paul POULAIN <paul.poulain(a)free.fr>: > > ë�c�� a ��crit : > > > Before going further I was confused with Koha on > > > "MARC tag structure admin for default MARC framework". > > > > > > The label is in French only. Could I change it to English? > > > > During install, you have choosen "UNIMARC". UNIMARC tables are in > > french only, where MARC21 tables are in english only. > > Can I load UNIMARC English table up for Koha? > > We, China and Taiwan library authorities, do not use MARC21 > for non-English materials. > > I hate to tell you that most of Chinese librarian can not > understand French. > > > -- > Best regards > > ë�c��, Anthony Mao, mao at lins dot fju dot edu dot tw > ICQ 30941847 > Department of Library and Information Science > http://www.lins.fju.edu.tw/mao/index.htm > TEL +886 2 29052334, FAX +886 2 29017405 > Fu-jen Catholic University, Taipei, Taiwan, R.O.C. >

4 3

Koha World Map
by 毛慶禎 12 Mar '06

12 Mar '06

Hi, Please kindly tell me what happen with Koha World Map? It was at http://koha.org/about/map/ several months ago. -- Best regards 毛慶禎, Anthony Mao, mao at lins dot fju dot edu dot tw ICQ 30941847 Department of Library and Information Science http://www.lins.fju.edu.tw/mao/index.htm TEL +886 2 29052334, FAX +886 2 29017405 Fu-jen Catholic University, Taipei, Taiwan, R.O.C.

2 2

Building zebradb
by Joshua Ferraro 12 Mar '06

12 Mar '06

Tumer, Sebastian's been good enough to respond to your post (I forwarded this to the koha-zebra list). If you get a change, could you join koha-zebra (if you're not already on it) and follow up -- I've a feeling it could prove to be a very productive thread. Cheers, Joshua ----- Forwarded message from Sebastian Hammer <quinn(a)indexdata.com> ----- Hi Joshua, Thanks for this feedback, it's very interesting. Clearly some of the issues you describe (i.e. a lack of stability around upadets) indicate software problems, but there's also some interesting ideas for possible refinements or new developments which I think would be really useful to get into the general development plans for the software... there are more folks at ID involved in Zebra development that I'd like to get into these thoughts... I dunno if a wiki or just a larger zebra-dev list is in order, but it's something to think about. Joshua Ferraro wrote: >----- Forwarded message from Tümer Garip <tgarip(a)neu.edu.tr> ----- > > >Hi, >We have now put the zebra into production level systems. So here is some >experience to share. > >Building the zebra database from single records is a veeeeery looong >process. (100K records 150k items) > > Yes, that confirms my expectations. We could think about building some kind of buffering into for first-time updating, or else the logic has to be in the application, as you've seen... the situation is particularly grim if shadow-indexing is enabled during indexing and every record is committed, since this causes a sync of the disks, which could take up to a whole second. Also, I'm not sure which version of Zebra you're using? I've been foing some performance-testing of Zebra for the Internet Archive, and noted quite a difference between 1.3 and 1.4 (the CVS version) which is really where all the development happens. >Best method we found: > >1- Change zebra.cfg file to include > >iso2079.recordType:grs.marcxml.collection >recordType:grs.xml.collection > >2- Write (or hack export.pl) to export all the marc records as one big >chunk to the correct directory with an extension .iso2079 And system >call "zebraidx -g iso2079 -d <dbnamehere> update records -n". > >This ensures that zebra knows its reading marc records rather than xml >and builds 100K+ records in zooming speed. >Your zoom module always uses the grs.xml filter while you can anytime >update or reindex any big chunk of the database as long as you have marc >records. > > Good strategy, I think.. but of course it's weird and awkward to have to use two different formats, especially when they both have limitations. We really must look into handling ISO2709 from ZOOM. Mind you, version 1.4 should be able to read multiple collection-wrapped MARCXML records in one file, but only (AFAIK) in conjunction witht the new XSLT-based index rules. I *would* like to try to develop a good way to work with bibliographic data in that framework. >3-We are still using the old API so we read the xml and use >MARC::Record->new_from_xml( $xmldata ) >A note here that we did not had to upgrade MARC::Record or MARC::Charset >at all. Any marc created within KOHA is UTF8 and any marc imported into >KOHA (old marc_subfield_tables) was correctly decoded to utf8 with >char_decode of biblio. > > Why not ask for the records in ISO2709 from Zebra if that's what you want to work with (when records are loaded in MARCXML, it can spit out either 2709 or XML)? Or is it just that you want to have them in MARC::Record? At any rate, ISO2709 is a more compact exchange format, so it seems to make sense to use it. >4- We modified circ2.pm and items table to have item onloan field and >mapped it to marc holdings data. Now our opac search do not call mysql >but for the branchname. > >5- Average updates per day is about 2000 (circulation+cataloger). I can >say that the speed of the zoom search which slows down during a commit >operation is acceptable considering the speed gain we have on the >search. > > Again, I'd be keen to know which Zebra version you're running? Beause the Internet Archive will be doing similar point-updates for records (only a *lot* more often than 2000 times per day) I have been looking a lot at the update speed for these small changes that only affect a single term in a record (like a circulation code).. In my test database of 150K records, 10K average size, I get update times around .5-.7 seconds, which just seems intuitively faster than it should have to be. I'm going to nudge the coders to see if we can possibly do this better. >6- Zebra behaves very well with searches but is very tempremental with >updates. A queue of updates sometimes crashes the zebraserver. When the >database crash we can not save anything even though we are using shadow >files. I'll be reporting on this issue once we can isolate the problems. > > Now *that* is nasty. Do you have *any* way of consistently recreating the problem? Even if you don't, I'm sure Adam and the Zebra crew would like to see some stack traces of those crashes!! --Sebastian >Regards, >Tumer > > > >_______________________________________________ >Koha-devel mailing list >Koha-devel(a)nongnu.org >http://lists.nongnu.org/mailman/listinfo/koha-devel > >----- End forwarded message ----- > > > -- Sebastian Hammer, Index Data quinn(a)indexdata.com www.indexdata.com Ph: (603) 209-6853 ----- End forwarded message ----- -- Joshua Ferraro VENDOR SERVICES FOR OPEN-SOURCE SOFTWARE President, Technology migration, training, maintenance, support LibLime Featuring Koha Open-Source ILS jmf(a)liblime.com |Full Demos at http://liblime.com/koha |1(888)KohaILS

1 0

reports/reservereport and circ/reserve.pl duplicate
by Paul POULAIN 10 Mar '06

10 Mar '06

Hi, i'm working on future 2.2.6 and i'm wondering if reports/reservereport.pl (commited by joshua, coming from katipo) and circ/reserve.pl (from hdl) were not the same ? (circ/reserve being more complete from my point of view, and without anything hardcoded) I think yes. (login test/test) http://bureau.paulpoulain.com:9007/cgi-bin/koha/reports/reservereport.pl and : http://bureau.paulpoulain.com:9007/cgi-bin/koha/circ/reserve.pl -- Paul POULAIN et Henri Damien LAURENT Consultants indépendants en logiciels libres et bibliothéconomie (http://www.koha-fr.org)

3 2