Fwd: RE: rebuild_zebra and a limit at 700 000 biblios
Scott, I already have : register: /home/koha/var/lib/zebradb/biblios/register:40G shadow: /home/koha/var/lib/zebradb/biblios/shadow:40G or I'm looking at the wrong file... double check ... unless i'm blind (or really too tired), I have 40G, so that should be enough :\ -------- Message original -------- Sujet: RE: [Koha-devel] rebuild_zebra and a limit at 700 000 biblios Date : Thu, 15 Jul 2010 17:17:16 -0400 De : Scott Kushner <skushner@mtpl.org> Pour : Paul Poulain <paul.poulain@biblibre.com> Paul, I think you have to increase your memory buffer in biblios.cfg to 8G or more. I had that problem. # Lock File Area lockDir: /root/koha-dev/var/lock/zebradb/biblios perm.anonymous:ar perm.root:rw passwd: /root/koha-dev/etc/zebradb/etc/passwd register: /root/koha-dev/var/lib/zebradb/biblios/register:8G shadow: /root/koha-dev/var/lib/zebradb/biblios/shadow:8G # Temp File area for result sets setTmpDir: /root/koha-dev/var/lib/zebradb/biblios/tmp # Temp File area for index program keyTmpDir: /root/koha-dev/var/lib/zebradb/biblios/key # Approx. Memory usage during indexing memMax: 150M rank:rank-1 truncmax: 1000000000 Scott Kushner Information Technologies Middletown Public Library -----Original Message----- From: koha-devel-bounces@lists.koha-community.org [mailto:koha-devel-bounces@lists.koha-community.org] On Behalf Of Paul Poulain Sent: Thursday, July 15, 2010 5:11 PM To: koha-devel@lists.koha-community.org Subject: [Koha-devel] rebuild_zebra and a limit at 700 000 biblios Hello, We have a database of 726k biblios and 1.2M items. When we rebuild_zebra, everything is OK on the export. Then, the rebuild_zebra seems OK as well, except that only 702k biblios are indexed. We don't have any error, everything seems OK. Does anyone have seen this problem ? Logs : exporting biblio ==================== Records exported: 726210 <snip> 06:43:19-08/07 zebraidx(314) [log] Records: 700000 i/u/d 700000/0/0 06:43:29-08/07 zebraidx(314) [log] Records: 701000 i/u/d 701000/0/0 06:43:38-08/07 zebraidx(314) [log] Records: 702000 i/u/d 702000/0/0 07:42:38-08/07 zebraidx(314) [log] Merge 0% <snip> 07:00:52-08/07 zebraidx(314) [log] Merge 95.4% completed; 48 seconds remaining 07:01:02-08/07 zebraidx(314) [log] Merge 96.4% completed; 37 seconds remaining 07:01:12-08/07 zebraidx(314) [log] Merge 97.4% completed; 26 seconds remaining 07:01:22-08/07 zebraidx(314) [log] Merge 98.7% completed; 13 seconds remaining 07:01:32-08/07 zebraidx(314) [log] Merge 99.9% completed; 0 seconds remaining 07:01:38-08/07 zebraidx(314) [log] Iterations: isam/dict 301420236/21553295 07:01:38-08/07 zebraidx(314) [log] Dict: inserts/updates/deletions: 21553295/0/0 07:01:42-08/07 zebraidx(314) [log] Records: 702954 i/u/d 702954/0/0 07:01:42-08/07 zebraidx(314) [log] zebra_stop: 20557.86 6312.50 1447.43 Search works on every biblio, except ones after 702955. -- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc Tel : (33) 4 91 81 35 08 _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
Paul, I find that around 70,000 biblios starts to push th 4GB limit, so 700,000 may start to push the 40GB limit, but from what you've pasted, you're not in that situation (there'd be a warn). Do you have enough hard drive space total for the exported files, register and shadow? Maybe it's that you can't export all the files? This is a shot in the dark, really. -Ian 2010/7/15 Paul Poulain <paul.poulain@biblibre.com>
Scott,
I already have : register: /home/koha/var/lib/zebradb/biblios/register:40G shadow: /home/koha/var/lib/zebradb/biblios/shadow:40G
or I'm looking at the wrong file... double check ... unless i'm blind (or really too tired), I have 40G, so that should be enough :\
-------- Message original -------- Sujet: RE: [Koha-devel] rebuild_zebra and a limit at 700 000 biblios Date : Thu, 15 Jul 2010 17:17:16 -0400 De : Scott Kushner <skushner@mtpl.org> <skushner@mtpl.org> Pour : Paul Poulain <paul.poulain@biblibre.com> <paul.poulain@biblibre.com>
Paul, I think you have to increase your memory buffer in biblios.cfg to 8G or more. I had that problem.
# Lock File Area lockDir: /root/koha-dev/var/lock/zebradb/biblios perm.anonymous:ar perm.root:rw passwd: /root/koha-dev/etc/zebradb/etc/passwd register: /root/koha-dev/var/lib/zebradb/biblios/register:8G shadow: /root/koha-dev/var/lib/zebradb/biblios/shadow:8G
# Temp File area for result sets setTmpDir: /root/koha-dev/var/lib/zebradb/biblios/tmp
# Temp File area for index program keyTmpDir: /root/koha-dev/var/lib/zebradb/biblios/key
# Approx. Memory usage during indexing memMax: 150M rank:rank-1 truncmax: 1000000000
Scott Kushner Information Technologies Middletown Public Library
-----Original Message----- From: koha-devel-bounces@lists.koha-community.org
[mailto:koha-devel-bounces@lists.koha-community.org <koha-devel-bounces@lists.koha-community.org>] On Behalf Of Paul Poulain Sent: Thursday, July 15, 2010 5:11 PM To: koha-devel@lists.koha-community.org Subject: [Koha-devel] rebuild_zebra and a limit at 700 000 biblios
Hello,
We have a database of 726k biblios and 1.2M items. When we rebuild_zebra, everything is OK on the export. Then, the rebuild_zebra seems OK as well, except that only 702k biblios are indexed. We don't have any error, everything seems OK.
Does anyone have seen this problem ?
Logs : exporting biblio ==================== Records exported: 726210 <snip> 06:43:19-08/07 zebraidx(314) [log] Records: 700000 i/u/d 700000/0/0 06:43:29-08/07 zebraidx(314) [log] Records: 701000 i/u/d 701000/0/0 06:43:38-08/07 zebraidx(314) [log] Records: 702000 i/u/d 702000/0/0 07:42:38-08/07 zebraidx(314) [log] Merge 0% <snip> 07:00:52-08/07 zebraidx(314) [log] Merge 95.4% completed; 48 seconds remaining 07:01:02-08/07 zebraidx(314) [log] Merge 96.4% completed; 37 seconds remaining 07:01:12-08/07 zebraidx(314) [log] Merge 97.4% completed; 26 seconds remaining 07:01:22-08/07 zebraidx(314) [log] Merge 98.7% completed; 13 seconds remaining 07:01:32-08/07 zebraidx(314) [log] Merge 99.9% completed; 0 seconds remaining 07:01:38-08/07 zebraidx(314) [log] Iterations: isam/dict 301420236/21553295 07:01:38-08/07 zebraidx(314) [log] Dict: inserts/updates/deletions: 21553295/0/0 07:01:42-08/07 zebraidx(314) [log] Records: 702954 i/u/d 702954/0/0 07:01:42-08/07 zebraidx(314) [log] zebra_stop: 20557.86 6312.50 1447.43
Search works on every biblio, except ones after 702955.
-- Paul POULAINhttp://www.biblibre.com Expert en Logiciels Libres pour l'info-doc Tel : (33) 4 91 81 35 08
_______________________________________________ Koha-devel mailing listKoha-devel@lists.koha-community.orghttp://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
-- Ian Walls Lead Development Specialist ByWater Solutions Phone # (888) 900-8944 http://bywatersolutions.com ian.walls@bywatersolutions.com Twitter: @sekjal
2010/7/15 Ian Walls <ian.walls@bywatersolutions.com>
Paul,
I find that around 70,000 biblios starts to push th 4GB limit, so 700,000 may start to push the 40GB limit, but from what you've pasted, you're not in that situation (there'd be a warn).
FWIW, Zebra claims to be able to handle 100GB of record data. http://www.indexdata.com/zebra/doc/features.html#features-scalability
The zebraidx program would fail loudly with a descriptive error if you exceeded either the configured allocation or available disk space. Some thing else is going on here. I would first confirm that all the expected records are actually in the export file. Clay 2010/7/15 Ian Walls <ian.walls@bywatersolutions.com>
Paul,
I find that around 70,000 biblios starts to push th 4GB limit, so 700,000 may start to push the 40GB limit, but from what you've pasted, you're not in that situation (there'd be a warn).
Do you have enough hard drive space total for the exported files, register and shadow? Maybe it's that you can't export all the files? This is a shot in the dark, really.
-Ian
2010/7/15 Paul Poulain <paul.poulain@biblibre.com>
Scott,
I already have : register: /home/koha/var/lib/zebradb/biblios/register:40G shadow: /home/koha/var/lib/zebradb/biblios/shadow:40G
or I'm looking at the wrong file... double check ... unless i'm blind (or really too tired), I have 40G, so that should be enough :\
-------- Message original -------- Sujet: RE: [Koha-devel] rebuild_zebra and a limit at 700 000 biblios Date : Thu, 15 Jul 2010 17:17:16 -0400 De : Scott Kushner <skushner@mtpl.org> <skushner@mtpl.org> Pour : Paul Poulain <paul.poulain@biblibre.com> <paul.poulain@biblibre.com>
Paul, I think you have to increase your memory buffer in biblios.cfg to 8G or more. I had that problem.
# Lock File Area lockDir: /root/koha-dev/var/lock/zebradb/biblios perm.anonymous:ar perm.root:rw passwd: /root/koha-dev/etc/zebradb/etc/passwd register: /root/koha-dev/var/lib/zebradb/biblios/register:8G shadow: /root/koha-dev/var/lib/zebradb/biblios/shadow:8G
# Temp File area for result sets setTmpDir: /root/koha-dev/var/lib/zebradb/biblios/tmp
# Temp File area for index program keyTmpDir: /root/koha-dev/var/lib/zebradb/biblios/key
# Approx. Memory usage during indexing memMax: 150M rank:rank-1 truncmax: 1000000000
Scott Kushner Information Technologies Middletown Public Library
-----Original Message----- From: koha-devel-bounces@lists.koha-community.org
[mailto:koha-devel-bounces@lists.koha-community.org <koha-devel-bounces@lists.koha-community.org>] On Behalf Of Paul Poulain Sent: Thursday, July 15, 2010 5:11 PM To: koha-devel@lists.koha-community.org Subject: [Koha-devel] rebuild_zebra and a limit at 700 000 biblios
Hello,
We have a database of 726k biblios and 1.2M items. When we rebuild_zebra, everything is OK on the export. Then, the rebuild_zebra seems OK as well, except that only 702k biblios are indexed. We don't have any error, everything seems OK.
Does anyone have seen this problem ?
Logs : exporting biblio ==================== Records exported: 726210 <snip> 06:43:19-08/07 zebraidx(314) [log] Records: 700000 i/u/d 700000/0/0 06:43:29-08/07 zebraidx(314) [log] Records: 701000 i/u/d 701000/0/0 06:43:38-08/07 zebraidx(314) [log] Records: 702000 i/u/d 702000/0/0 07:42:38-08/07 zebraidx(314) [log] Merge 0% <snip> 07:00:52-08/07 zebraidx(314) [log] Merge 95.4% completed; 48 seconds remaining 07:01:02-08/07 zebraidx(314) [log] Merge 96.4% completed; 37 seconds remaining 07:01:12-08/07 zebraidx(314) [log] Merge 97.4% completed; 26 seconds remaining 07:01:22-08/07 zebraidx(314) [log] Merge 98.7% completed; 13 seconds remaining 07:01:32-08/07 zebraidx(314) [log] Merge 99.9% completed; 0 seconds remaining 07:01:38-08/07 zebraidx(314) [log] Iterations: isam/dict 301420236/21553295 07:01:38-08/07 zebraidx(314) [log] Dict: inserts/updates/deletions: 21553295/0/0 07:01:42-08/07 zebraidx(314) [log] Records: 702954 i/u/d 702954/0/0 07:01:42-08/07 zebraidx(314) [log] zebra_stop: 20557.86 6312.50 1447.43
Search works on every biblio, except ones after 702955.
-- Paul POULAINhttp://www.biblibre.com Expert en Logiciels Libres pour l'info-doc Tel : (33) 4 91 81 35 08
_______________________________________________ Koha-devel mailing listKoha-devel@lists.koha-community.orghttp://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
-- Ian Walls Lead Development Specialist ByWater Solutions Phone # (888) 900-8944 http://bywatersolutions.com ian.walls@bywatersolutions.com Twitter: @sekjal
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
I have a 300 000 biblios and it works fine with 16G. I propose a trick to count exported bibios from the database : 1. Use -x in rebuild_zebra.pl to export in xml 2. Use -k to keep exported file 3. Run rebuild_zebra.pl 4. Go into export folder : /tmp/XXX/biblio 5. Run this command : cat exported_records | grep "<datafield tag=\"090\"" | wc -l Assuming koha biblionumber is stored in field 090. Good luck. 2010/7/15 Fouts, Clay <cfouts@ptfs.com>
The zebraidx program would fail loudly with a descriptive error if you exceeded either the configured allocation or available disk space. Some thing else is going on here.
I would first confirm that all the expected records are actually in the export file.
Clay
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
-- Fridolyn SOMERS Information and Communication Technologies engineer +33(0)6.86.63.77.68 fridolyn.somers@gmail.com
participants (5)
-
Chris Nighswonger -
Fouts, Clay -
Fridolyn SOMERS -
Ian Walls -
Paul Poulain