I've just been reading Bug 2453 - (very) large biblio/item handling; it appears to have received no comments since 2010-05-21. If I understand the situation correctly (from Paul Poulain's comments) if a biblio/items record exceeds 100Kb then zebra may fail. He suggests: [snip] 2 possibilities : - fix Koha to limit the size of a biblio/items (check that we don't ge more than 100kB) - fix the export during rebuild_zebra. I think the 1st solution is the best. I am currently defining our serials cataloguing policy - choices: one biblio for perhaps 1,200 items; one biblio per item (probably unmanageable in OPAC); or one biblio per year|volume. Currently the third choice is preferred, one (perhaps repetitive) biblio with up to 12 items, occasionally 52 (we do not handle dailies.) However, even at twelve items per biblio we will often go beyond 100Kb; we are a research library where abstracts, indexes and keywords are, by definition, fairly voluminous and are the "bread and butter" of our users. Should this bug prompt me to perhaps rethink the policy? Has 3.4 changed this situation? Is there anything else in the pipeline that I should be aware of? Many thanks for any guidance, Paul Tired old sys-admin
On 22 April 2011 09:44, Paul <paul.a@aandc.org> wrote:
I've just been reading Bug 2453 - (very) large biblio/item handling; it appears to have received no comments since 2010-05-21. If I understand the situation correctly (from Paul Poulain's comments) if a biblio/items record exceeds 100Kb then zebra may fail. He suggests:
[snip] 2 possibilities : - fix Koha to limit the size of a biblio/items (check that we don't ge more than 100kB) - fix the export during rebuild_zebra. I think the 1st solution is the best.
I am currently defining our serials cataloguing policy - choices: one biblio for perhaps 1,200 items; one biblio per item (probably unmanageable in OPAC); or one biblio per year|volume. Currently the third choice is preferred, one (perhaps repetitive) biblio with up to 12 items, occasionally 52 (we do not handle dailies.)
However, even at twelve items per biblio we will often go beyond 100Kb; we are a research library where abstracts, indexes and keywords are, by definition, fairly voluminous and are the "bread and butter" of our users. Should this bug prompt me to perhaps rethink the policy? Has 3.4 changed this situation? Is there anything else in the pipeline that I should be aware of?
If you run rebuild_zebra.pl with the -x switch (just for biblio -x doesnt work with authorities) this limitation does not exist. I suggest running 2 rebuild_zebra.pl calls one for biblios rebuild_zebra.pl -x -b -z and one for authorities rebuild_zebra.pl -a -z Chris
Chris, I don't know if this changes in 3.4, because of the feature listed in the release_notes_3_4_0.txt about moving items out of biblioitems (line 151). But the problem we had with records with large numbers of items wasn't just in Zebra, it was in the overall size of the MARC record exceeding the 99999 record length. Does removing the items from biblioitems help? Thanks. Linda Culberson On 4/21/2011 4:47 PM, Chris Cormack wrote:
On 22 April 2011 09:44, Paul<paul.a@aandc.org> wrote:
I've just been reading Bug 2453 - (very) large biblio/item handling; it appears to have received no comments since 2010-05-21. If I understand the situation correctly (from Paul Poulain's comments) if a biblio/items record exceeds 100Kb then zebra may fail. He suggests:
[snip] 2 possibilities : - fix Koha to limit the size of a biblio/items (check that we don't ge more than 100kB) - fix the export during rebuild_zebra. I think the 1st solution is the best.
I am currently defining our serials cataloguing policy - choices: one biblio for perhaps 1,200 items; one biblio per item (probably unmanageable in OPAC); or one biblio per year|volume. Currently the third choice is preferred, one (perhaps repetitive) biblio with up to 12 items, occasionally 52 (we do not handle dailies.)
However, even at twelve items per biblio we will often go beyond 100Kb; we are a research library where abstracts, indexes and keywords are, by definition, fairly voluminous and are the "bread and butter" of our users. Should this bug prompt me to perhaps rethink the policy? Â Has 3.4 changed this situation? Â Is there anything else in the pipeline that I should be aware of?
If you run rebuild_zebra.pl with the -x switch (just for biblio -x doesnt work with authorities) this limitation does not exist.
I suggest running 2 rebuild_zebra.pl calls one for biblios rebuild_zebra.pl -x -b -z and one for authorities rebuild_zebra.pl -a -z
Chris _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Linda Culberson lculber@mdah.state.ms.us Archives and Records Services Division Ms. Dept. of Archives & History P. O. Box 571 Jackson, MS 39205-0571 Telephone: 601/576-6873 Facsimile: 601/576-6824
On 22 April 2011 09:57, Linda Culberson <lculber@mdah.state.ms.us> wrote:
Chris, I don't know if this changes in 3.4, because of the feature listed in the release_notes_3_4_0.txt about moving items out of biblioitems (line 151). But the problem we had with records with large numbers of items wasn't just in Zebra, it was in the overall size of the MARC record exceeding the 99999 record length.
Hi Linda Yes it was the size of the record exceeding that, that meant when it was being exported as iso2709 it was truncated, and broke zebra. If you export it as xml (-x) then this limitation doesn't happen. The only other place I am aware of, that this might be a problem was if you wanted to export all your records as iso2709 for some other reason.
Does removing the items from biblioitems help?
Yes and No, it speeds everything up, but when you go to index, you still need to give zebra the entire record, which without -x will cause it to be truncated on export. Hope this helps Chris
Thank you both Chris and Linda. We were also intending to use a couple of spare subfields in 952 for these keywords and abstracts. I'm now wondering if the [default?] mapping to $unlinked_item_subfields will exceed the 99999 limit? I'll set up a dummy set of entries tomorrow, with more than 10Kb in the item abstracts, and more than 100Kb in the biblio/items and give it a spin. Resetting the cron job for re-indexing zebra will probably present no difficulties (set to every minute at the moment.) Chris: very small point, a typo? at line 120 of release_notes_3_4_0.txt -- 'like" should be "link"? Thanks and best regards, Paul At 04:57 PM 4/21/2011 -0500, Linda Culberson wrote:
Chris, I don't know if this changes in 3.4, because of the feature listed in the release_notes_3_4_0.txt about moving items out of biblioitems (line 151). But the problem we had with records with large numbers of items wasn't just in Zebra, it was in the overall size of the MARC record exceeding the 99999 record length.
Does removing the items from biblioitems help?
Thanks. Linda Culberson
On 4/21/2011 4:47 PM, Chris Cormack wrote:
On 22 April 2011 09:44, Paul<paul.a@aandc.org> wrote:
I've just been reading Bug 2453 - (very) large biblio/item handling; it appears to have received no comments since 2010-05-21. If I understand the situation correctly (from Paul Poulain's comments) if a biblio/items record exceeds 100Kb then zebra may fail. He suggests:
[snip] 2 possibilities : - fix Koha to limit the size of a biblio/items (check that we don't ge more than 100kB) - fix the export during rebuild_zebra. I think the 1st solution is the best.
I am currently defining our serials cataloguing policy - choices: one biblio for perhaps 1,200 items; one biblio per item (probably unmanageable in OPAC); or one biblio per year|volume. Currently the third choice is preferred, one (perhaps repetitive) biblio with up to 12 items, occasionally 52 (we do not handle dailies.)
However, even at twelve items per biblio we will often go beyond 100Kb; we are a research library where abstracts, indexes and keywords are, by definition, fairly voluminous and are the "bread and butter" of our users. Should this bug prompt me to perhaps rethink the policy? Â Has 3.4 changed this situation? Â Is there anything else in the pipeline that I should be aware of? If you run rebuild_zebra.pl with the -x switch (just for biblio -x doesnt work with authorities) this limitation does not exist.
I suggest running 2 rebuild_zebra.pl calls one for biblios rebuild_zebra.pl -x -b -z and one for authorities rebuild_zebra.pl -a -z
Chris _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Linda Culberson lculber@mdah.state.ms.us Archives and Records Services Division Ms. Dept. of Archives & History P. O. Box 571 Jackson, MS 39205-0571 Telephone: 601/576-6873 Facsimile: 601/576-6824
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
On 22 April 2011 09:44, Paul <paul.a@aandc.org> wrote:
I've just been reading Bug 2453 - (very) large biblio/item handling; it appears to have received no comments since 2010-05-21. If I understand the situation correctly (from Paul Poulain's comments) if a biblio/items record exceeds 100Kb then zebra may fail. He suggests:
[snip] 2 possibilities : - fix Koha to limit the size of a biblio/items (check that we don't ge more than 100kB) - fix the export during rebuild_zebra. I think the 1st solution is the best.
I am currently defining our serials cataloguing policy - choices: one biblio for perhaps 1,200 items; one biblio per item (probably unmanageable in OPAC); or one biblio per year|volume. Currently the third choice is preferred, one (perhaps repetitive) biblio with up to 12 items, occasionally 52 (we do not handle dailies.)
However, even at twelve items per biblio we will often go beyond 100Kb; we are a research library where abstracts, indexes and keywords are, by definition, fairly voluminous and are the "bread and butter" of our users. Should this bug prompt me to perhaps rethink the policy? Has 3.4 changed this situation? Is there anything else in the pipeline that I should be aware of?
If you run rebuild_zebra.pl with the -x switch (just for biblio -x doesnt work with authorities) this limitation does not exist. Well, strictly speaking, you are right, with -x, the whole information will be indexed. But, since koha internally takes only iso2709 from zebra and not working
Le 21/04/2011 23:47, Chris Cormack a écrit : directly from marcxml then if your iso2709 is big... and your biblionumber is at the end of your iso2709, then you will not be able to get the biblionumber and therefore not be able to get the record. And then, limiting the quantity of items indexed can be a workaround. my 2 cents.
I suggest running 2 rebuild_zebra.pl calls one for biblios rebuild_zebra.pl -x -b -z and one for authorities rebuild_zebra.pl -a -z
Chris
Henri-Damien LAURENT
participants (4)
-
Chris Cormack -
LAURENT Henri-Damien -
Linda Culberson -
Paul