[Koha-devel] Fwd: RE: rebuild_zebra and a limit at 700 000 biblios

[Fouts, Clay]

The zebraidx program would fail loudly with a descriptive error if you
exceeded either the configured allocation or available disk space. Some
thing else is going on here.

I would first confirm that all the expected records are actually in the
export file.

Clay


2010/7/15 Ian Walls <ian.walls at bywatersolutions.com>

> Paul,
>
>
> I find that around 70,000 biblios starts to push th 4GB limit, so 700,000
> may start to push the 40GB limit, but from what you've pasted, you're not in
> that situation (there'd be a warn).
>
> Do you have enough hard drive space total for the exported files, register
> and shadow?  Maybe it's that you can't export all the files?  This is a shot
> in the dark, really.
>
>
> -Ian
>
> 2010/7/15 Paul Poulain <paul.poulain at biblibre.com>
>
>  Scott,
>>
>> I already have :
>> register: /home/koha/var/lib/zebradb/biblios/register:40G
>> shadow: /home/koha/var/lib/zebradb/biblios/shadow:40G
>>
>> or I'm looking at the wrong file...
>> double check ...
>> unless i'm blind (or really too tired), I have 40G, so that should be
>> enough :\
>>
>> -------- Message original --------  Sujet: RE: [Koha-devel] rebuild_zebra
>> and a limit at 700 000 biblios  Date : Thu, 15 Jul 2010 17:17:16 -0400  De :
>> Scott Kushner <skushner at mtpl.org> <skushner at mtpl.org>  Pour : Paul
>> Poulain <paul.poulain at biblibre.com> <paul.poulain at biblibre.com>
>>
>> Paul,
>> I think you have to increase your memory buffer in biblios.cfg to 8G or
>> more. I had that problem.
>>
>> # Lock File Area
>> lockDir: /root/koha-dev/var/lock/zebradb/biblios
>> perm.anonymous:ar
>> perm.root:rw
>> passwd: /root/koha-dev/etc/zebradb/etc/passwd
>> register: /root/koha-dev/var/lib/zebradb/biblios/register:8G
>> shadow: /root/koha-dev/var/lib/zebradb/biblios/shadow:8G
>>
>> # Temp File area for result sets
>> setTmpDir: /root/koha-dev/var/lib/zebradb/biblios/tmp
>>
>> # Temp File area for index program
>> keyTmpDir: /root/koha-dev/var/lib/zebradb/biblios/key
>>
>> # Approx. Memory usage during indexing
>> memMax: 150M
>> rank:rank-1
>> truncmax: 1000000000
>>
>>
>> Scott Kushner
>> Information Technologies
>> Middletown Public Library
>>
>> -----Original Message-----
>> From: koha-devel-bounces at lists.koha-community.org
>>
>> [mailto:koha-devel-bounces at lists.koha-community.org <koha-devel-bounces at lists.koha-community.org>] On Behalf Of Paul
>> Poulain
>> Sent: Thursday, July 15, 2010 5:11 PM
>> To: koha-devel at lists.koha-community.org
>> Subject: [Koha-devel] rebuild_zebra and a limit at 700 000 biblios
>>
>> Hello,
>>
>> We have a database of 726k biblios and 1.2M items.
>> When we rebuild_zebra, everything is OK on the export.
>> Then, the rebuild_zebra seems OK as well, except that only 702k biblios
>> are indexed.
>> We don't have any error, everything seems OK.
>>
>> Does anyone have seen this problem ?
>>
>> Logs :
>> exporting biblio
>> ====================
>> Records exported: 726210
>> <snip>
>> 06:43:19-08/07 zebraidx(314) [log] Records: 700000 i/u/d 700000/0/0
>> 06:43:29-08/07 zebraidx(314) [log] Records: 701000 i/u/d 701000/0/0
>> 06:43:38-08/07 zebraidx(314) [log] Records: 702000 i/u/d 702000/0/0
>> 07:42:38-08/07 zebraidx(314) [log] Merge 0% <snip>
>> 07:00:52-08/07 zebraidx(314) [log] Merge 95.4% completed; 48 seconds
>> remaining
>> 07:01:02-08/07 zebraidx(314) [log] Merge 96.4% completed; 37 seconds
>> remaining
>> 07:01:12-08/07 zebraidx(314) [log] Merge 97.4% completed; 26 seconds
>> remaining
>> 07:01:22-08/07 zebraidx(314) [log] Merge 98.7% completed; 13 seconds
>> remaining
>> 07:01:32-08/07 zebraidx(314) [log] Merge 99.9% completed; 0 seconds
>> remaining
>> 07:01:38-08/07 zebraidx(314) [log] Iterations: isam/dict
>> 301420236/21553295
>> 07:01:38-08/07 zebraidx(314) [log] Dict: inserts/updates/deletions:
>> 21553295/0/0
>> 07:01:42-08/07 zebraidx(314) [log] Records: 702954 i/u/d 702954/0/0
>> 07:01:42-08/07 zebraidx(314) [log] zebra_stop: 20557.86 6312.50 1447.43
>>
>> Search works on every biblio, except ones after 702955.
>>
>>
>> --
>> Paul POULAINhttp://www.biblibre.com
>> Expert en Logiciels Libres pour l'info-doc
>> Tel : (33) 4 91 81 35 08
>>
>> _______________________________________________
>> Koha-devel mailing listKoha-devel at lists.koha-community.orghttp://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
>>
>>
>> _______________________________________________
>> Koha-devel mailing list
>> Koha-devel at lists.koha-community.org
>> http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
>>
>
>
>
> --
> Ian Walls
> Lead Development Specialist
> ByWater Solutions
> Phone # (888) 900-8944
> http://bywatersolutions.com
> ian.walls at bywatersolutions.com
> Twitter: @sekjal
>
> _______________________________________________
> Koha-devel mailing list
> Koha-devel at lists.koha-community.org
> http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/koha-devel/attachments/20100715/60c981da/attachment.htm>