(sort of) Refactoring rebuild_zebra
Hi everyone, I'm writing a replacement for zebraqueue-daemon.pl so that we can set the reindexing interval for authorities (and biblios) via systempreferences and we can set intervals in seconds (for our impatient librarians) wich cannot be acknowleged using cron jobs. I'll be posting the script soon, but I've been thinking of kind-of refactoring rebuild_zebra into a reusable library so that the daemon doesn't spawn a perl interpreter every (for example) 30 seconds. A question related to the topic is: why we don't trigger the update whenever a cataloguer click "save" for a biblio or auth record? Perhaps this refactoring serves for an implementation of this. Any suggestions? Opinions? Thanks To+
On 20 August 2010 05:00, Tomas Cohen Arazi <tomascohen@gmail.com> wrote:
Hi everyone, I'm writing a replacement for zebraqueue-daemon.pl so that we can set the reindexing interval for authorities (and biblios) via systempreferences and we can set intervals in seconds (for our impatient librarians) wich cannot be acknowleged using cron jobs. I'll be posting the script soon, but I've been thinking of kind-of refactoring rebuild_zebra into a reusable library so that the daemon doesn't spawn a perl interpreter every (for example) 30 seconds.
A question related to the topic is: why we don't trigger the update whenever a cataloguer click "save" for a biblio or auth record? Perhaps this refactoring serves for an implementation of this.
Because it's slow. Thats why it is done as a seperate process, the index needs to be updated every time an item is issued or returned as well as when it is changed by a cataloguer. It would slow circulation down way too much. Chris
I wonder now if we could seperate the two. So cataloging updated zebra immediately, but circulation was queued. Would that work? On Fri, 2010-08-20 at 06:53 +1200, Chris Cormack wrote:
On 20 August 2010 05:00, Tomas Cohen Arazi <tomascohen@gmail.com> wrote:
Hi everyone, I'm writing a replacement for zebraqueue-daemon.pl so that we can set the reindexing interval for authorities (and biblios) via systempreferences and we can set intervals in seconds (for our impatient librarians) wich cannot be acknowleged using cron jobs. I'll be posting the script soon, but I've been thinking of kind-of refactoring rebuild_zebra into a reusable library so that the daemon doesn't spawn a perl interpreter every (for example) 30 seconds.
A question related to the topic is: why we don't trigger the update whenever a cataloguer click "save" for a biblio or auth record? Perhaps this refactoring serves for an implementation of this.
Because it's slow. Thats why it is done as a seperate process, the index needs to be updated every time an item is issued or returned as well as when it is changed by a cataloguer. It would slow circulation down way too much.
Chris _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
-- Michael Hafen Systems Analyst and Programmer Washington County School District Utah, USA for Koha checkout http://development.washk12.org/gitweb/ or git://development.washk12.org/koha
On 20 August 2010 09:39, Michael Hafen <mdhafen@tech.washk12.org> wrote:
I wonder now if we could seperate the two. So cataloging updated zebra immediately, but circulation was queued.
Would that work?
Its certainly a possibility, would involve a bit of work, since the calls are down in module level, not script level, and at that level often the code doesn't know whether it's being called as a result of circulation, or a cataloguing change, (or a branch transfer .. or whatever else i've forgotten :-)) But if we were careful, worst that could happen is we would reindex something that had already been reindexed. Chris
Chris Cormack wrote:
On 20 August 2010 09:39, Michael Hafen <mdhafen@tech.washk12.org> wrote:
I wonder now if we could seperate the two. So cataloging updated zebra immediately, but circulation was queued.
Would that work?
Its certainly a possibility, would involve a bit of work, since the calls are down in module level, not script level, and at that level often the code doesn't know whether it's being called as a result of circulation, or a cataloguing change, (or a branch transfer .. or whatever else i've forgotten :-))
This would be good because I sometimes get asked if Koha's index does "realtime" updating and at the moment the honest answer depends how you define "realtime". Unlike other systems, librarians can see for themselves the index update process. But I don't know if this limits adoption, as so far no library has offered to pay the co-op to change it. ;-) I have a slight concern about the rebuild_zebra.pl rewrite: is it perpetuating the daemon/cron competing implementations? Can we figure out one size that fits all? Maybe have one daemon running and then different ways for a cron job or librarian interface task to signal (some SIG perhaps?) that it needs to act? Hope that helps, -- MJ Ray (slef) Webmaster and developer for hire at | software www.software.coop http://mjr.towers.org.uk | .... co IMO only: see http://mjr.towers.org.uk/email.html | .... op
On Tue, Aug 24, 2010 at 2:02 PM, MJ Ray <mjr@phonecoop.coop> wrote:
Chris Cormack wrote:
On 20 August 2010 09:39, Michael Hafen <mdhafen@tech.washk12.org> wrote:
I wonder now if we could seperate the two. So cataloging updated zebra immediately, but circulation was queued.
Would that work?
Its certainly a possibility, would involve a bit of work, since the calls are down in module level, not script level, and at that level often the code doesn't know whether it's being called as a result of circulation, or a cataloguing change, (or a branch transfer .. or whatever else i've forgotten :-))
This would be good because I sometimes get asked if Koha's index does "realtime" updating and at the moment the honest answer depends how you define "realtime". Unlike other systems, librarians can see for themselves the index update process.
But I don't know if this limits adoption, as so far no library has offered to pay the co-op to change it. ;-)
I have a slight concern about the rebuild_zebra.pl rewrite: is it perpetuating the daemon/cron competing implementations? Can we figure out one size that fits all? Maybe have one daemon running and then different ways for a cron job or librarian interface task to signal (some SIG perhaps?) that it needs to act?
I've just send a rewrite of zebraqueue daemon that does run (sort of refactoring of) rebuild_zebra as one would do with cron. I think is the right way to handle this. Avoid using cron, and configure the frequency of the update via systempreferences. I first thought of a script in bash but had some concerns about the system load as each pass would spawn a new perl process. To+
Tomas Cohen Arazi wrote:
On Tue, Aug 24, 2010 at 2:02 PM, MJ Ray <mjr@phonecoop.coop> wrote:
we figure out one size that fits all? Maybe have one daemon running and then different ways for a cron job or librarian interface task to signal (some SIG perhaps?) that it needs to act?
I've just send a rewrite of zebraqueue daemon that does run (sort of refactoring of) rebuild_zebra as one would do with cron. I think is the right way to handle this. Avoid using cron, and configure the frequency of the update via systempreferences.
As you know, I've made a few comments on that and having a daemon around does open the door to realtime reindexing, but I don't think that rewrite does that yet. I don't think it would be difficult to allow the librarian interface to wake it up sooner than the update frequency set in systempreferences, would it? This has been discussed over on the koha list and one case where realtime reindexing would really help is when authorities are being added at cataloguing-time. At the moment, some librarians are changing their workflow to suit Koha, which seems backwards to me. Thanks, -- MJ Ray (slef), member of www.software.coop, a for-more-than-profit co-op. Webmaster, Debian Developer, Past Koha RM, statistician, former lecturer. In My Opinion Only: see http://mjr.towers.org.uk/email.html Available for hire for Koha work http://www.software.coop/products/koha
On Thu, Aug 26, 2010 at 7:45 AM, MJ Ray <mjr@phonecoop.coop> wrote:
Tomas Cohen Arazi wrote:
On Tue, Aug 24, 2010 at 2:02 PM, MJ Ray <mjr@phonecoop.coop> wrote:
we figure out one size that fits all? Maybe have one daemon running and then different ways for a cron job or librarian interface task to signal (some SIG perhaps?) that it needs to act?
I've just send a rewrite of zebraqueue daemon that does run (sort of refactoring of) rebuild_zebra as one would do with cron. I think is the right way to handle this. Avoid using cron, and configure the frequency of the update via systempreferences.
As you know, I've made a few comments on that and having a daemon around does open the door to realtime reindexing, but I don't think that rewrite does that yet. I don't think it would be difficult to allow the librarian interface to wake it up sooner than the update frequency set in systempreferences, would it?
That's exactly what I explained in the IRC. The new zebraqueue-daemon script is just a workaround (running that script too often seems worse than triggering it instantly on updates). As I said, this could be acheived directly from the librarian interface and that's why I'm proposing a library for rebuild_zebra tasks. My main concern is about atomicity and possible race conditions. We need to atomice: 1) Read zebraqueue table and make a list of records that need updating. 2) Generate the files for that records that will be passed to zebraidx 3) Run zebraidx on those files 4) Mark those records in zebraqueue as done. We need to avoid spawning a new update until this is finished, concurrently being able to add new rows to zebraqueue table for concurrent cataloguing.
This has been discussed over on the koha list and one case where realtime reindexing would really help is when authorities are being added at cataloguing-time. At the moment, some librarians are changing their workflow to suit Koha, which seems backwards to me.
I agree with that as my librarians express that on an hourly basis :-D To+
Le 24/08/2010 19:02, MJ Ray a écrit : > Chris Cormack wrote: >> On 20 August 2010 09:39, Michael Hafen <mdhafen@tech.washk12.org> wrote: >>> I wonder now if we could seperate the two. So cataloging updated zebra >>> immediately, but circulation was queued. >>> >>> Would that work? >>> >> Its certainly a possibility, would involve a bit of work, since the >> calls are down in module level, not script level, and at that level >> often the code doesn't know whether it's being called as a result of >> circulation, or a cataloguing change, (or a branch transfer .. or >> whatever else i've forgotten :-)) > > This would be good because I sometimes get asked if Koha's index does > "realtime" updating and at the moment the honest answer depends how > you define "realtime". Unlike other systems, librarians can see for > themselves the index update process. > > But I don't know if this limits adoption, as so far no library has > offered to pay the co-op to change it. ;-) > > I have a slight concern about the rebuild_zebra.pl rewrite: is it > perpetuating the daemon/cron competing implementations? Can > we figure out one size that fits all? Maybe have one daemon > running and then different ways for a cron job or librarian > interface task to signal (some SIG perhaps?) that it needs to act? > > Hope that helps, If I may contribute my 2 cents : I think that rebuild zebra should be separated into two parts : - export of biblio - indexation We could export biblio into a directory, always the same and then process that with inotify. In fact I think we need a more powerful and robust export tool. And since zebraidx is used in rebuild zebra we could use inotify to process that and get the logs, to tell whether the indexation went good or bad. Then it would not be possible to launch more than one zebraidx at once and crash the machine. It would be safer since we could have more little pieces of information to index. And we would not depend on a cronjob but on a deamon. Drawback is that it would only run on Linux.... at the moment... Maybe we could use jnotify... My 2 cents. -- Henri-Damien LAURENT
On Tue, Aug 24, 2010 at 2:29 PM, LAURENT Henri-Damien <laurenthdl@alinto.com> wrote: > Le 24/08/2010 19:02, MJ Ray a écrit : >> Chris Cormack wrote: >>> On 20 August 2010 09:39, Michael Hafen <mdhafen@tech.washk12.org> wrote: >>>> I wonder now if we could seperate the two. So cataloging updated zebra >>>> immediately, but circulation was queued. >>>> >>>> Would that work? >>>> >>> Its certainly a possibility, would involve a bit of work, since the >>> calls are down in module level, not script level, and at that level >>> often the code doesn't know whether it's being called as a result of >>> circulation, or a cataloguing change, (or a branch transfer .. or >>> whatever else i've forgotten :-)) >> >> This would be good because I sometimes get asked if Koha's index does >> "realtime" updating and at the moment the honest answer depends how >> you define "realtime". Unlike other systems, librarians can see for >> themselves the index update process. >> >> But I don't know if this limits adoption, as so far no library has >> offered to pay the co-op to change it. ;-) >> >> I have a slight concern about the rebuild_zebra.pl rewrite: is it >> perpetuating the daemon/cron competing implementations? Can >> we figure out one size that fits all? Maybe have one daemon >> running and then different ways for a cron job or librarian >> interface task to signal (some SIG perhaps?) that it needs to act? >> >> Hope that helps, > If I may contribute my 2 cents : > I think that rebuild zebra should be separated into two parts : > - export of biblio > - indexation > We could export biblio into a directory, always the same and then > process that with inotify. > In fact I think we need a more powerful and robust export tool. > And since zebraidx is used in rebuild zebra we could use inotify to > process that and get the logs, to tell whether the indexation went good > or bad. > Then it would not be possible to launch more than one zebraidx at once > and crash the machine. My proposal of using a daemon that handles the 'sleep task' inside of it tries to avoid that race condition as it is intended to replace the cronjob. To+
LAURENT Henri-Damien <laurenthdl@alinto.com> wrote: [...]
We could export biblio into a directory, always the same and then process that with inotify. In fact I think we need a more powerful and robust export tool. And since zebraidx is used in rebuild zebra we could use inotify to process that and get the logs, to tell whether the indexation went good or bad. [...]
I'm guessing this is reasonably accurate: "inotify is a Linux kernel subsystem that provides file system event notification. It was written by John McCutchan with help from Robert Love and later Amy Griffis to replace dnotify. It was included in the mainline kernel from release 2.6.13 (June 18, 2005), and could be compiled into 2.6.12 and possibly earlier releases by use of a patch..." http://a.vu/w:Inotify Why inotify and not sending a signal to the daemon? Signals would run more widely than only Linux (which I don't think I need right now, but after some time trying to get Koha running well on MacOS, I don't want to sour anyone else's well). Puzzled, -- MJ Ray (slef), member of www.software.coop, a for-more-than-profit co-op. Webmaster, Debian Developer, Past Koha RM, statistician, former lecturer. In My Opinion Only: see http://mjr.towers.org.uk/email.html Available for hire for Koha work http://www.software.coop/products/koha
participants (5)
-
Chris Cormack -
LAURENT Henri-Damien -
Michael Hafen -
MJ Ray -
Tomas Cohen Arazi