zebraop commented [for joshua 1st]
Hi again, Joshua, you told me yesterday that the commenting of zebraop (*) in dev_week was not a bug, but due to a problem with _zconn in Koha (that looses the connection we don't know why) + it seems that updates are long on large databases. Thus, for adding/updating biblios/items, NPL uses update_items script, running once a day (and thus having a zebra DB uptodate only once a day, which can be considered as a shame, even if not blocking) SAN-OP will go live in january, 3rd, and will do large test (library closed to public, everybody working/testing koha) on dec, 14th, to I thought it could be a good idea to investigate the problem during those tests. Joshua told me that NPL uses zebra 1.x, and SAN-OP will use zebra 2.0, so maybe it could solve the problem. Here is my question : how do we know that the update/insert went wrong ? an internal server error ? an error in the log ? nothing, just being unable to find the added/modified biblio/item ? Did the problem occur once a week or once every hour ? did they notice it occurs more on heavy load or any particular situation ? I'll report here the result of SAN-OP tests. Hoping zebra 2.x will solve it. (*) zebraop is the sub that is called when a biblio/item is added/updated/modified. -- Paul POULAIN et Henri Damien LAURENT Consultants indépendants en logiciels libres et bibliothéconomie (http://www.koha-fr.org) Tel : 04 91 31 45 19
Paul POULAIN a écrit :
Hi again,
Joshua, you told me yesterday that the commenting of zebraop (*) in dev_week was not a bug, but due to a problem with _zconn in Koha (that looses the connection we don't know why) + it seems that updates are long on large databases.
Thus, for adding/updating biblios/items, NPL uses update_items script, running once a day (and thus having a zebra DB uptodate only once a day, which can be considered as a shame, even if not blocking)
SAN-OP will go live in january, 3rd, and will do large test (library closed to public, everybody working/testing koha) on dec, 14th, to I thought it could be a good idea to investigate the problem during those tests.
Joshua told me that NPL uses zebra 1.x, and SAN-OP will use zebra 2.0, so maybe it could solve the problem.
Here is my question : how do we know that the update/insert went wrong ? an internal server error ? an error in the log ? nothing, just being unable to find the added/modified biblio/item ? Did the problem occur once a week or once every hour ? did they notice it occurs more on heavy load or any particular situation ?
I'll report here the result of SAN-OP tests. Hoping zebra 2.x will solve it.
(*) zebraop is the sub that is called when a biblio/item is added/updated/modified. Some update performance on large database issues were raised quite recently on zebralist. It is latest yaz version and zebra can be part of the solution. It is also said it improves stability. -- Henri-Damien LAURENT
On Wed, Dec 06, 2006 at 12:07:13PM +0100, Paul POULAIN wrote:
Joshua, you told me yesterday that the commenting of zebraop (*) in dev_week was not a bug, but due to a problem with _zconn in Koha (that looses the connection we don't know why) + it seems that updates are long on large databases.
Thus, for adding/updating biblios/items, NPL uses update_items script, running once a day (and thus having a zebra DB uptodate only once a day, which can be considered as a shame, even if not blocking)
SAN-OP will go live in january, 3rd, and will do large test (library closed to public, everybody working/testing koha) on dec, 14th, to I thought it could be a good idea to investigate the problem during those tests.
Joshua told me that NPL uses zebra 1.x, and SAN-OP will use zebra 2.0, so maybe it could solve the problem.
Here is my question : how do we know that the update/insert went wrong ? an internal server error ? an error in the log ? nothing, just being unable to find the added/modified biblio/item ? The errors I've seen in the past occur when a zebraop is executed (an add, edit, or delete operation on the zebra db over ZOOM). So just to clarify what I've tested, before NPL went live, I did some simple testing with zebraop running for all circ operations (to update statuses of items) and cataloging (add, edit, delete). Given that circ is so frequent (many times per second), I discovered that Zebra could not keep up (very slow), and so created update_items.pl, which I supposed would run every hour or less to update the item statuses.
After NPL went live, we had some problems with the index crashing frequently. The crash wasn't too much trouble as searching still worked ... just edits/adds/deletes didn't. So my solution was to comment out zebraop alltogether and only use update_items.pl to update the index. Even update_items.pl would fail occassionally, especially if run frequently during the day (once per hour for instance), so now it runs only once per day :/.
Did the problem occur once a week or once every hour ? did they notice it occurs more on heavy load or any particular situation ? It's almost impossible to keep track of when it happens, especially with a 7 branch library that does over half a million circ annually ... so unfortunately I can't say for sure what triggers an index crash. I do know that ID has fixed several bugs in Zebra 1.x based on reports I've sent with log files, etc., and that Zebra hasn't crashed in over two weeks. I plan to try hourly runs of update_item.pl again soon ... I will report my findings when I do.
Hope that helps. Cheers, -- Joshua Ferraro SUPPORT FOR OPEN-SOURCE SOFTWARE President, Technology migration, training, maintenance, support LibLime Featuring Koha Open-Source ILS jmf@liblime.com |Full Demos at http://liblime.com/koha |1(888)KohaILS
Hi Paul, The issue of ZEBRA updates being slow does not get resolved in zebra 2.0 either. However if you check code in head for zebraop and zebraopserver you will see that the only solution (and working without error) i found is not to update the zebra but to write a new table zebraqueue the biblionumber,server,actiontotake. Another process called zebraqueue_start in z3950 folder runs full time as a deamon and checks this table at set times. Time interval is set at preferences 'zebrawait' in seconds. Currently mine is set to 30 seconds and with around 2000 updates a day zebra can cope with it. On very busy periods changing the system pref to 10 minutes can save the circulation desk but i only had to do that once in 2 months. Regarding zebra updates and zebra crashes we have not been getting any crashes since we stopped using shadow files(contrary to what zebra documentation says). I can say that this method of asynchronous zebra update and no shadow files is working and you should look into implementing it in rel_3 as well before SUN-OP goes live. Cheers, Tümer -----Original Message----- From: koha-devel-bounces+tgarip=neu.edu.tr@nongnu.org [mailto:koha-devel-bounces+tgarip=neu.edu.tr@nongnu.org] On Behalf Of Paul POULAIN Sent: Wednesday, December 06, 2006 1:07 PM To: Koha-devel@nongnu.org Cc: Jérôme Pouchol Subject: [Koha-devel] zebraop commented [for joshua 1st] Hi again, Joshua, you told me yesterday that the commenting of zebraop (*) in dev_week was not a bug, but due to a problem with _zconn in Koha (that looses the connection we don't know why) + it seems that updates are long on large databases. Thus, for adding/updating biblios/items, NPL uses update_items script, running once a day (and thus having a zebra DB uptodate only once a day, which can be considered as a shame, even if not blocking) SAN-OP will go live in january, 3rd, and will do large test (library closed to public, everybody working/testing koha) on dec, 14th, to I thought it could be a good idea to investigate the problem during those tests. Joshua told me that NPL uses zebra 1.x, and SAN-OP will use zebra 2.0, so maybe it could solve the problem. Here is my question : how do we know that the update/insert went wrong ? an internal server error ? an error in the log ? nothing, just being unable to find the added/modified biblio/item ? Did the problem occur once a week or once every hour ? did they notice it occurs more on heavy load or any particular situation ? I'll report here the result of SAN-OP tests. Hoping zebra 2.x will solve it. (*) zebraop is the sub that is called when a biblio/item is added/updated/modified. -- Paul POULAIN et Henri Damien LAURENT Consultants indépendants en logiciels libres et bibliothéconomie (http://www.koha-fr.org) Tel : 04 91 31 45 19 _______________________________________________ Koha-devel mailing list Koha-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/koha-devel
participants (4)
-
Henri-Damien LAURENT -
Joshua Ferraro -
Paul POULAIN -
Tümer Garip