Hi, We have now put the zebra into production level systems. So here is some experience to share. Building the zebra database from single records is a veeeeery looong process. (100K records 150k items) Best method we found: 1- Change zebra.cfg file to include iso2079.recordType:grs.marcxml.collection recordType:grs.xml.collection 2- Write (or hack export.pl) to export all the marc records as one big chunk to the correct directory with an extension .iso2079 And system call "zebraidx -g iso2079 -d <dbnamehere> update records -n". This ensures that zebra knows its reading marc records rather than xml and builds 100K+ records in zooming speed. Your zoom module always uses the grs.xml filter while you can anytime update or reindex any big chunk of the database as long as you have marc records. 3-We are still using the old API so we read the xml and use MARC::Record->new_from_xml( $xmldata ) A note here that we did not had to upgrade MARC::Record or MARC::Charset at all. Any marc created within KOHA is UTF8 and any marc imported into KOHA (old marc_subfield_tables) was correctly decoded to utf8 with char_decode of biblio. 4- We modified circ2.pm and items table to have item onloan field and mapped it to marc holdings data. Now our opac search do not call mysql but for the branchname. 5- Average updates per day is about 2000 (circulation+cataloger). I can say that the speed of the zoom search which slows down during a commit operation is acceptable considering the speed gain we have on the search. 6- Zebra behaves very well with searches but is very tempremental with updates. A queue of updates sometimes crashes the zebraserver. When the database crash we can not save anything even though we are using shadow files. I'll be reporting on this issue once we can isolate the problems. Regards, Tumer
Tümer Garip a écrit :
Hi,
Hello Tümer,
We have now put the zebra into production level systems. So here is some experience to share. Building the zebra database from single records is a veeeeery looong process. (100K records 150k items)
Best method we found:
1- Change zebra.cfg file to include
iso2079.recordType:grs.marcxml.collection recordType:grs.xml.collection if I understand, you now have 2 types of records in your DB (or 2 differents representations of a record)
2- Write (or hack export.pl) to export all the marc records as one big chunk to the correct directory with an extension .iso2079 And system call "zebraidx -g iso2079 -d <dbnamehere> update records -n".
Could you send us the code for export.pl ?
This ensures that zebra knows its reading marc records rather than xml and builds 100K+ records in zooming speed. Your zoom module always uses the grs.xml filter while you can anytime update or reindex any big chunk of the database as long as you have marc records.
Great, I think I understand.
3-We are still using the old API weso read the xml and use MARC::Record->new_from_xml( $xmldata ) A note here that we did not had to upgrade MARC::Record or MARC::Charset at all. Any marc created within KOHA is UTF8 and any marc imported into KOHA (old marc_subfield_tables) was correctly decoded to utf8 with char_decode of biblio.
Could it be possible to use this zebra.cfg to manage iso2709 through Perl-ZOOM ? If yes, we could avoid marc => xml => zoom and zoom => xml => marc transformations.
4- We modified circ2.pm and items table to have item onloan field and mapped it to marc holdings data. Now our opac search do not call mysql but for the branchname.
Could you send us/me the code too ?
5- Average updates per day is about 2000 (circulation+cataloger). I can say that the speed of the zoom search which slows down during a commit operation is acceptable considering the speed gain we have on the search.
6- Zebra behaves very well with searches but is very tempremental with updates. A queue of updates sometimes crashes the zebraserver. When the database crash we can not save anything even though we are using shadow files. I'll be reporting on this issue once we can isolate the problems.
You're definetly a gem too ;-) -- Paul POULAIN et Henri Damien LAURENT Consultants indépendants en logiciels libres et bibliothéconomie (http://www.koha-fr.org)
Hi Paul, The script you have requested I have posted them tou you directly as I donno whether the list accepts attachments. Regarding your questions below: 1- Well yes I am feeding zebra with 2 different kinds of records (iso2079 and xml) and I do not see any problem with this as ZEBRA changes everything to its own format anyway. This way I can create a 100K+ records in around 5 minutes in ZEBRA (this time excludes the time it takes to export my whole database whisch is around 15 min). I can use ZOOM with XML to update - delete or add new records. 2-Regarding whether we can use perl-ZOOM with iso2079 records it seems at the moment NO!. And it is all our wish that indexdata does incorporate this facility for us at some stage. Best of luck, Tumer -----Original Message----- From: Paul POULAIN [mailto:paul.poulain@free.fr] Sent: Wednesday, March 15, 2006 7:19 PM To: Tümer Garip Cc: koha-devel@nongnu.org; koha-zebra@nongnu.org Subject: Re: [Koha-devel] Building zebradb Tümer Garip a écrit :
Hi,
Hello Tümer,
We have now put the zebra into production level systems. So here is some experience to share. Building the zebra database from single records is a veeeeery looong process. (100K records 150k items)
Best method we found:
1- Change zebra.cfg file to include
iso2079.recordType:grs.marcxml.collection recordType:grs.xml.collection if I understand, you now have 2 types of records in your DB (or 2 differents representations of a record)
2- Write (or hack export.pl) to export all the marc records as one big
chunk to the correct directory with an extension .iso2079 And system call "zebraidx -g iso2079 -d <dbnamehere> update records -n".
Could you send us the code for export.pl ?
This ensures that zebra knows its reading marc records rather than xml
and builds 100K+ records in zooming speed. Your zoom module always uses the grs.xml filter while you can anytime update or reindex any big chunk of the database as long as you have marc records.
Great, I think I understand.
3-We are still using the old API weso read the xml and use MARC::Record->new_from_xml( $xmldata ) A note here that we did not had
to upgrade MARC::Record or MARC::Charset at all. Any marc created within KOHA is UTF8 and any marc imported into KOHA (old marc_subfield_tables) was correctly decoded to utf8 with char_decode of biblio.
Could it be possible to use this zebra.cfg to manage iso2709 through Perl-ZOOM ? If yes, we could avoid marc => xml => zoom and zoom => xml => marc transformations.
4- We modified circ2.pm and items table to have item onloan field and mapped it to marc holdings data. Now our opac search do not call mysql
but for the branchname.
Could you send us/me the code too ?
5- Average updates per day is about 2000 (circulation+cataloger). I can say that the speed of the zoom search which slows down during a commit operation is acceptable considering the speed gain we have on the search.
6- Zebra behaves very well with searches but is very tempremental with
updates. A queue of updates sometimes crashes the zebraserver. When the database crash we can not save anything even though we are using shadow files. I'll be reporting on this issue once we can isolate the problems.
You're definetly a gem too ;-) -- Paul POULAIN et Henri Damien LAURENT Consultants indépendants en logiciels libres et bibliothéconomie (http://www.koha-fr.org)
Hi all, I don't know whether everything is set for v3 KOHA but here are some food for thought. In our library we use LC classification and callnumbering. 1- For this the item call number is constructed from MARC21 field 050a and 050b plus ant other local addition. KOHA has a system preference for putting only one subfield into itemcallnumber. Couldn't we have it so that if we write 050ab it actually parses and reads both subfields or maybe even better 050a,XXXb for real flexibility. 2- Inventory reports are supposed to order the books as they appear on the shelf. For us (LC) order goes like this: AB1.5 C25 2004 AB1.5 C203 2004 AB5 .C203 2004 AB41 .C203 2004 AB234 .C203 2004 (just samples not real callnumbers) This sorting is a litte complex that letters are sorted as text and numbers as numbers. While KOHA sorts them as a text field and we have AB1.5 C203 2004 AB1.5 C25 2004 AB234 .C203 2004 AB41 .C203 2004 AB5 .C203 2004 To be able to achieve the LC sort order we had to write a parsing routine to item callnumber and add 5 more fields to items table. If this is an issue for others as well may be we can incorporate it in official release with a system preference whether to use this sorting or not. The code is already with Paul (in biblio.pm) but we need to tweak it and for that I'll ask for some help on coding if its usefull to others as well. I need to know whether it is useful to you so we can start tweaking this module which is not on our priorities list. Regards, Tumer
On Thu, Mar 16, 2006 at 05:52:12PM +0200, Tümer Garip wrote:
Hi all, I don't know whether everything is set for v3 KOHA but here are some food for thought. No, not set completely yet.
In our library we use LC classification and callnumbering.
1- For this the item call number is constructed from MARC21 field 050a and 050b plus ant other local addition. KOHA has a system preference for putting only one subfield into itemcallnumber. Couldn't we have it so that if we write 050ab it actually parses and reads both subfields or maybe even better 050a,XXXb for real flexibility. In fact, aren't there cases where a library uses two or more different item call numbers for different collections? For instance, NPL uses Dewey for non-fiction but fiction uses a locally-constructed method.
So in addition to needing to parse multiple subfields it should also support multiple mappings ... Any suggestions for how to handle that?
2- Inventory reports are supposed to order the books as they appear on the shelf. For us (LC) order goes like this:
AB1.5 C25 2004 AB1.5 C203 2004 AB5 .C203 2004 AB41 .C203 2004 AB234 .C203 2004 (just samples not real callnumbers) This sorting is a litte complex that letters are sorted as text and numbers as numbers.
While KOHA sorts them as a text field and we have AB1.5 C203 2004 AB1.5 C25 2004 AB234 .C203 2004 AB41 .C203 2004 AB5 .C203 2004
To be able to achieve the LC sort order we had to write a parsing routine to item callnumber and add 5 more fields to items table.
If this is an issue for others as well may be we can incorporate it in official release with a system preference whether to use this sorting or not. Yes!!! Please do.
I need to know whether it is useful to you so we can start tweaking this module which is not on our priorities list. What kinds of tweaking would be necessary?
Cheers, -- Joshua Ferraro VENDOR SERVICES FOR OPEN-SOURCE SOFTWARE President, Technology migration, training, maintenance, support LibLime Featuring Koha Open-Source ILS jmf@liblime.com |Full Demos at http://liblime.com/koha |1(888)KohaILS
Hi, Joshua said:
In fact, aren't there cases where a library uses two or more different item call numbers for different collections? For instance, NPL uses Dewey for non-fiction >>but fiction uses a locally-constructed method.
So in addition to needing to parse multiple subfields it should also support multiple mappings ... Any suggestions for how to handle that? Well I am not so sure of what you ment but if you mean parsing 050a,050b,090b,245c altogether to form the itemcalnumber from whichever combination there is i.e just combine them in order as long as there is something in these fields thats easy (we can write that). If not explain what the need is in layman terms as we dont know what you use.
What kinds of tweaking would be necessary? Well I think the code can be written more intuitevly rather than assuming that the cataloger have put say the dot in the correct place or not. All the fields are doubles while some can be left as text etc. The code does work by the way just that it need more thought for flexibility
Cheers, Tumer Garip NEU Libraries & Information Systems Director -----Original Message----- From: Joshua Ferraro [mailto:jmf@liblime.com] Sent: Thursday, March 16, 2006 6:01 PM To: Tümer Garip Cc: 'Paul POULAIN'; koha-devel@nongnu.org Subject: Re: [Koha-devel] Some script ideas on version 3 On Thu, Mar 16, 2006 at 05:52:12PM +0200, Tümer Garip wrote:
Hi all, I don't know whether everything is set for v3 KOHA but here are some food for thought. No, not set completely yet.
In our library we use LC classification and callnumbering.
1- For this the item call number is constructed from MARC21 field 050a
and 050b plus ant other local addition. KOHA has a system preference for putting only one subfield into itemcallnumber. Couldn't we have it
so that if we write 050ab it actually parses and reads both subfields or maybe even better 050a,XXXb for real flexibility. In fact, aren't there cases where a library uses two or more different item call numbers for different collections? For instance, NPL uses Dewey for non-fiction but fiction uses a locally-constructed method.
So in addition to needing to parse multiple subfields it should also support multiple mappings ... Any suggestions for how to handle that?
2- Inventory reports are supposed to order the books as they appear on
the shelf. For us (LC) order goes like this:
AB1.5 C25 2004 AB1.5 C203 2004 AB5 .C203 2004 AB41 .C203 2004 AB234 .C203 2004 (just samples not real callnumbers) This sorting is a litte complex that letters are sorted as text and numbers as numbers.
While KOHA sorts them as a text field and we have AB1.5 C203 2004 AB1.5 C25 2004 AB234 .C203 2004 AB41 .C203 2004 AB5 .C203 2004
To be able to achieve the LC sort order we had to write a parsing routine to item callnumber and add 5 more fields to items table.
If this is an issue for others as well may be we can incorporate it in
official release with a system preference whether to use this sorting or not. Yes!!! Please do.
I need to know whether it is useful to you so we can start tweaking this module which is not on our priorities list. What kinds of tweaking would be necessary?
Cheers, -- Joshua Ferraro VENDOR SERVICES FOR OPEN-SOURCE SOFTWARE President, Technology migration, training, maintenance, support LibLime Featuring Koha Open-Source ILS jmf@liblime.com |Full Demos at http://liblime.com/koha |1(888)KohaILS
Hi everyone, Our future customer would like to have Interlibrary Loan (ILL). I saw few exchanges on the mailing list that was sent few years ago. I want to know if anything was integrated in Koha because I couldn't find anything related to ILL in Koha or in the roadmaps. If not, any advices about this type of feature? Thanks, Eric Bégin eric at gokoha dot com On another topic, we got word that Koha community agreed for not using Koha in the name of their company. We weren't aware of that when we picked GoKoha. As I mentioned to some of you, we will change our name before going public. Thanks for your understanding and thanks to Joshua for letting me know about that agreement.
Eric - Koha does have provisions for "Institutional" (organization) members, with different issuing rules and borrower details, which could be used to keep track of loans to other libraries. What precise capabilities is the customer seeking? Stephen Eric Bégin wrote:
Hi everyone,
Our future customer would like to have Interlibrary Loan (ILL).
I saw few exchanges on the mailing list that was sent few years ago. I want to know if anything was integrated in Koha because I couldn't find anything related to ILL in Koha or in the roadmaps.
If not, any advices about this type of feature?
Thanks,
Eric Bégin eric at gokoha dot com
-- Stephen Hedges Skemotah Solutions, USA www.skemotah.com -- shedges@skemotah.com
On Thu, Mar 16, 2006 at 11:48:30PM +0200, Tümer Garip wrote:
Hi, Joshua said:
In fact, aren't there cases where a library uses two or more different item call numbers for different collections? For instance, NPL uses Dewey for non-fiction >>but fiction uses a locally-constructed method.
So in addition to needing to parse multiple subfields it should also support multiple mappings ... Any suggestions for how to handle that? Well I am not so sure of what you ment but if you mean parsing 050a,050b,090b,245c altogether to form the itemcalnumber from whichever combination there is i.e just combine them in order as long as there is something in these fields thats easy (we can write that). If not explain what the need is in layman terms as we dont know what you use. What you describe is yet another function that would be nice to have but is a bit different than NPL's need :-). I will try to explain. NPL has two classification systems, one for non-fiction (Dewey) and one for fiction (two letter code for type of fiction followed by author's last name). You can view an example of this on NPL's catalog:
http://search.athenscounty.lib.oh.us/cgi-bin/koha/opac-detail.pl?bib=16595 In this case, 'SF' stands for 'science fiction'. Compare that call number with the more standard Dewey call number here: http://search.athenscounty.lib.oh.us/cgi-bin/koha/opac-detail.pl?bib=51475 This makes it quite difficult to do a search by call number or classification (NPL doesn't distinguish between these two). Does that make more sense? Any suggestions for how to handle this?
What kinds of tweaking would be necessary? Well I think the code can be written more intuitevly rather than assuming that the cataloger have put say the dot in the correct place or not. All the fields are doubles while some can be left as text etc. The code does work by the way just that it need more thought for flexibility Sounds like some good improvements ...
Sincerely, -- Joshua Ferraro VENDOR SERVICES FOR OPEN-SOURCE SOFTWARE President, Technology migration, training, maintenance, support LibLime Featuring Koha Open-Source ILS jmf@liblime.com |Full Demos at http://liblime.com/koha |1(888)KohaILS
participants (5)
-
Eric Bégin -
Joshua Ferraro -
Paul POULAIN -
Stephen Hedges -
Tümer Garip