Announcing an alternate indexing daemon for Zebra which is more memory efficient
Hi all, I've written a new indexing daemon for Zebra which you can find at https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27267. It's dead easy to use. Once the daemon is installed, you just need to update /etc/default/koha-common and you'll be using it. It only uses 6MB of RAM (instead of the 170-180MB I've seen rebuild_zebra.pl use in daemon mode), so it scales well on servers that host many instances of Koha. (For example, if you have 50 Koha instances on a server, you'll use 300MB instead of 9GB for idle Zebra indexing daemons.) That said, this indexing daemon does launch rebuild_zebra.pl in non-daemon mode when necessary, so you still need to have memory available to do actual work. Making rebuild_zebra.pl more memory efficient is another task in itself and not a pressing issue atm. I'm planning to use it shortly myself, but I figured that I would share it with everyone else, so that everyone else could benefit from it too. Hope others enjoy it as much as I do. David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595
Hello David, thanks for sharing, it looks great! We are going to test it on one of our servers. We'll report how things are going probably in few days... Josef On Wed, Jan 6, 2021 at 4:28 AM <dcook@prosentient.com.au> wrote:
Hi all,
I’ve written a new indexing daemon for Zebra which you can find at https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27267. It’s dead easy to use. Once the daemon is installed, you just need to update /etc/default/koha-common and you’ll be using it.
It only uses 6MB of RAM (instead of the 170-180MB I’ve seen rebuild_zebra.pl use in daemon mode), so it scales well on servers that host many instances of Koha. (For example, if you have 50 Koha instances on a server, you’ll use 300MB instead of 9GB for idle Zebra indexing daemons.) That said, this indexing daemon does launch rebuild_zebra.pl in non-daemon mode when necessary, so you still need to have memory available to do actual work. Making rebuild_zebra.pl more memory efficient is another task in itself and not a pressing issue atm.
I’m planning to use it shortly myself, but I figured that I would share it with everyone else, so that everyone else could benefit from it too.
Hope others enjoy it as much as I do.
David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
-- Josef Moravec josef.moravec@gmail.com
Oooh nice. Is there a protection against several launch ? Like in actual daemon --wait-for-lock ? Or is there no need ? Regards, Le 06/01/2021 à 08:50, Josef Moravec a écrit :
Hello David,
thanks for sharing, it looks great! We are going to test it on one of our servers. We'll report how things are going probably in few days...
Josef
On Wed, Jan 6, 2021 at 4:28 AM <dcook@prosentient.com.au <mailto:dcook@prosentient.com.au>> wrote:
Hi all,____
__ __
I’ve written a new indexing daemon for Zebra which you can find at https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27267 <https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27267>. It’s dead easy to use. Once the daemon is installed, you just need to update /etc/default/koha-common and you’ll be using it.____
__ __
It only uses 6MB of RAM (instead of the 170-180MB I’ve seen rebuild_zebra.pl <http://rebuild_zebra.pl> use in daemon mode), so it scales well on servers that host many instances of Koha. (For example, if you have 50 Koha instances on a server, you’ll use 300MB instead of 9GB for idle Zebra indexing daemons.) That said, this indexing daemon does launch rebuild_zebra.pl <http://rebuild_zebra.pl> in non-daemon mode when necessary, so you still need to have memory available to do actual work. Making rebuild_zebra.pl <http://rebuild_zebra.pl> more memory efficient is another task in itself and not a pressing issue atm.____
__ __
I’m planning to use it shortly myself, but I figured that I would share it with everyone else, so that everyone else could benefit from it too. ____
__ __
Hope others enjoy it as much as I do.____
__ __
David Cook____
Software Engineer____
Prosentient Systems____
Suite 7.03____
6a Glen St____
Milsons Point NSW 2061____
Australia____
__ __
Office: 02 9212 0899____
Online: 02 8005 0595____
__ __
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel <https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel> website : https://www.koha-community.org/ <https://www.koha-community.org/> git : https://git.koha-community.org/ <https://git.koha-community.org/> bugs : https://bugs.koha-community.org/ <https://bugs.koha-community.org/>
-- Josef Moravec josef.moravec@gmail.com <mailto:josef.moravec@gmail.com>
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
-- Fridolin SOMERS <fridolin.somers@biblibre.com> Software and system maintainer 🦄 BibLibre, France
There's no need for a couple reasons. First, all child process operations for my "zebra_indexer.pl" (ie checking for records to index and the actual indexing) cause the parent indexing daemon to block (by using waitpid($pid,0)) so there should be no overlapping runs. Second, even if multiple rebuild_zebra.pl processes were launched magically, the built-in locking in rebuild_zebra.pl would prevent any but the 1st process from running. So it's all good 😃 David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of Fridolin SOMERS Sent: Thursday, 7 January 2021 4:01 AM To: koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Announcing an alternate indexing daemon for Zebra which is more memory efficient Oooh nice. Is there a protection against several launch ? Like in actual daemon --wait-for-lock ? Or is there no need ? Regards, Le 06/01/2021 à 08:50, Josef Moravec a écrit :
Hello David,
thanks for sharing, it looks great! We are going to test it on one of our servers. We'll report how things are going probably in few days...
Josef
On Wed, Jan 6, 2021 at 4:28 AM <dcook@prosentient.com.au <mailto:dcook@prosentient.com.au>> wrote:
Hi all,____
__ __
I’ve written a new indexing daemon for Zebra which you can find at https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27267 <https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27267>. It’s dead easy to use. Once the daemon is installed, you just need to update /etc/default/koha-common and you’ll be using it.____
__ __
It only uses 6MB of RAM (instead of the 170-180MB I’ve seen rebuild_zebra.pl <http://rebuild_zebra.pl> use in daemon mode), so it scales well on servers that host many instances of Koha. (For example, if you have 50 Koha instances on a server, you’ll use 300MB instead of 9GB for idle Zebra indexing daemons.) That said, this indexing daemon does launch rebuild_zebra.pl <http://rebuild_zebra.pl> in non-daemon mode when necessary, so you still need to have memory available to do actual work. Making rebuild_zebra.pl <http://rebuild_zebra.pl> more memory efficient is another task in itself and not a pressing issue atm.____
__ __
I’m planning to use it shortly myself, but I figured that I would share it with everyone else, so that everyone else could benefit from it too. ____
__ __
Hope others enjoy it as much as I do.____
__ __
David Cook____
Software Engineer____
Prosentient Systems____
Suite 7.03____
6a Glen St____
Milsons Point NSW 2061____
Australia____
__ __
Office: 02 9212 0899____
Online: 02 8005 0595____
__ __
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel <https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel> website : https://www.koha-community.org/ <https://www.koha-community.org/> git : https://git.koha-community.org/ <https://git.koha-community.org/> bugs : https://bugs.koha-community.org/ <https://bugs.koha-community.org/>
-- Josef Moravec josef.moravec@gmail.com <mailto:josef.moravec@gmail.com>
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
-- Fridolin SOMERS <fridolin.somers@biblibre.com> Software and system maintainer 🦄 BibLibre, France _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
Ah of course, rebuild_zebra.pl itself has lock system. It ends if you dont use --wait-for-lock. Thanks ;) Le 07/01/2021 à 00:26, dcook@prosentient.com.au a écrit :
There's no need for a couple reasons.
First, all child process operations for my "zebra_indexer.pl" (ie checking for records to index and the actual indexing) cause the parent indexing daemon to block (by using waitpid($pid,0)) so there should be no overlapping runs.
Second, even if multiple rebuild_zebra.pl processes were launched magically, the built-in locking in rebuild_zebra.pl would prevent any but the 1st process from running.
So it's all good 😃
David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia
Office: 02 9212 0899 Online: 02 8005 0595
-----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of Fridolin SOMERS Sent: Thursday, 7 January 2021 4:01 AM To: koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Announcing an alternate indexing daemon for Zebra which is more memory efficient
Oooh nice.
Is there a protection against several launch ? Like in actual daemon --wait-for-lock ? Or is there no need ?
Regards,
Le 06/01/2021 à 08:50, Josef Moravec a écrit :
Hello David,
thanks for sharing, it looks great! We are going to test it on one of our servers. We'll report how things are going probably in few days...
Josef
On Wed, Jan 6, 2021 at 4:28 AM <dcook@prosentient.com.au <mailto:dcook@prosentient.com.au>> wrote:
Hi all,____
__ __
I’ve written a new indexing daemon for Zebra which you can find at https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27267 <https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27267>. It’s dead easy to use. Once the daemon is installed, you just need to update /etc/default/koha-common and you’ll be using it.____
__ __
It only uses 6MB of RAM (instead of the 170-180MB I’ve seen rebuild_zebra.pl <http://rebuild_zebra.pl> use in daemon mode), so it scales well on servers that host many instances of Koha. (For example, if you have 50 Koha instances on a server, you’ll use 300MB instead of 9GB for idle Zebra indexing daemons.) That said, this indexing daemon does launch rebuild_zebra.pl <http://rebuild_zebra.pl> in non-daemon mode when necessary, so you still need to have memory available to do actual work. Making rebuild_zebra.pl <http://rebuild_zebra.pl> more memory efficient is another task in itself and not a pressing issue atm.____
__ __
I’m planning to use it shortly myself, but I figured that I would share it with everyone else, so that everyone else could benefit from it too. ____
__ __
Hope others enjoy it as much as I do.____
__ __
David Cook____
Software Engineer____
Prosentient Systems____
Suite 7.03____
6a Glen St____
Milsons Point NSW 2061____
Australia____
__ __
Office: 02 9212 0899____
Online: 02 8005 0595____
__ __
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel <https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel> website : https://www.koha-community.org/ <https://www.koha-community.org/> git : https://git.koha-community.org/ <https://git.koha-community.org/> bugs : https://bugs.koha-community.org/ <https://bugs.koha-community.org/>
-- Josef Moravec josef.moravec@gmail.com <mailto:josef.moravec@gmail.com>
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
-- Fridolin SOMERS <fridolin.somers@biblibre.com> Software and system maintainer 🦄 BibLibre, France _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
-- Fridolin SOMERS <fridolin.somers@biblibre.com> Software and system maintainer 🦄 BibLibre, France
How are people going with this? I've been running it for about 1.5 months now with 0 complaints. As I just noted on 27267, I just implemented it on a small cloud server with only 4GB RAM and I was able to save 500MB switching to the alternative indexer. Quite a big win in a resource constrained environment! David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -----Original Message----- From: Fridolin SOMERS <fridolin.somers@biblibre.com> Sent: Thursday, 7 January 2021 7:08 PM To: dcook@prosentient.com.au; koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Announcing an alternate indexing daemon for Zebra which is more memory efficient Ah of course, rebuild_zebra.pl itself has lock system. It ends if you dont use --wait-for-lock. Thanks ;) Le 07/01/2021 à 00:26, dcook@prosentient.com.au a écrit :
There's no need for a couple reasons.
First, all child process operations for my "zebra_indexer.pl" (ie checking for records to index and the actual indexing) cause the parent indexing daemon to block (by using waitpid($pid,0)) so there should be no overlapping runs.
Second, even if multiple rebuild_zebra.pl processes were launched magically, the built-in locking in rebuild_zebra.pl would prevent any but the 1st process from running.
So it's all good 😃
David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia
Office: 02 9212 0899 Online: 02 8005 0595
-----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of Fridolin SOMERS Sent: Thursday, 7 January 2021 4:01 AM To: koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Announcing an alternate indexing daemon for Zebra which is more memory efficient
Oooh nice.
Is there a protection against several launch ? Like in actual daemon --wait-for-lock ? Or is there no need ?
Regards,
Le 06/01/2021 à 08:50, Josef Moravec a écrit :
Hello David,
thanks for sharing, it looks great! We are going to test it on one of our servers. We'll report how things are going probably in few days...
Josef
On Wed, Jan 6, 2021 at 4:28 AM <dcook@prosentient.com.au <mailto:dcook@prosentient.com.au>> wrote:
Hi all,____
__ __
I’ve written a new indexing daemon for Zebra which you can find at https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27267 <https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27267>. It’s dead easy to use. Once the daemon is installed, you just need to update /etc/default/koha-common and you’ll be using it.____
__ __
It only uses 6MB of RAM (instead of the 170-180MB I’ve seen rebuild_zebra.pl <http://rebuild_zebra.pl> use in daemon mode), so it scales well on servers that host many instances of Koha. (For example, if you have 50 Koha instances on a server, you’ll use 300MB instead of 9GB for idle Zebra indexing daemons.) That said, this indexing daemon does launch rebuild_zebra.pl <http://rebuild_zebra.pl> in non-daemon mode when necessary, so you still need to have memory available to do actual work. Making rebuild_zebra.pl <http://rebuild_zebra.pl> more memory efficient is another task in itself and not a pressing issue atm.____
__ __
I’m planning to use it shortly myself, but I figured that I would share it with everyone else, so that everyone else could benefit from it too. ____
__ __
Hope others enjoy it as much as I do.____
__ __
David Cook____
Software Engineer____
Prosentient Systems____
Suite 7.03____
6a Glen St____
Milsons Point NSW 2061____
Australia____
__ __
Office: 02 9212 0899____
Online: 02 8005 0595____
__ __
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel <https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel> website : https://www.koha-community.org/ <https://www.koha-community.org/> git : https://git.koha-community.org/ <https://git.koha-community.org/> bugs : https://bugs.koha-community.org/ <https://bugs.koha-community.org/>
-- Josef Moravec josef.moravec@gmail.com <mailto:josef.moravec@gmail.com>
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
-- Fridolin SOMERS <fridolin.somers@biblibre.com> Software and system maintainer 🦄 BibLibre, France _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
-- Fridolin SOMERS <fridolin.somers@biblibre.com> Software and system maintainer 🦄 BibLibre, France
Sounds great, Josef! I implemented it yesterday afternoon in production and so far I haven’t gotten any complaints. I’m also super pleased with the reduction in resource usage. Keen to hear your report! David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 From: Josef Moravec <josef.moravec@gmail.com> Sent: Wednesday, 6 January 2021 6:50 PM To: David Cook <dcook@prosentient.com.au> Cc: koha-devel <koha-devel@lists.koha-community.org> Subject: Re: [Koha-devel] Announcing an alternate indexing daemon for Zebra which is more memory efficient Hello David, thanks for sharing, it looks great! We are going to test it on one of our servers. We'll report how things are going probably in few days... Josef On Wed, Jan 6, 2021 at 4:28 AM <dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> > wrote: Hi all, I’ve written a new indexing daemon for Zebra which you can find at https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27267. It’s dead easy to use. Once the daemon is installed, you just need to update /etc/default/koha-common and you’ll be using it. It only uses 6MB of RAM (instead of the 170-180MB I’ve seen rebuild_zebra.pl <http://rebuild_zebra.pl> use in daemon mode), so it scales well on servers that host many instances of Koha. (For example, if you have 50 Koha instances on a server, you’ll use 300MB instead of 9GB for idle Zebra indexing daemons.) That said, this indexing daemon does launch rebuild_zebra.pl <http://rebuild_zebra.pl> in non-daemon mode when necessary, so you still need to have memory available to do actual work. Making rebuild_zebra.pl <http://rebuild_zebra.pl> more memory efficient is another task in itself and not a pressing issue atm. I’m planning to use it shortly myself, but I figured that I would share it with everyone else, so that everyone else could benefit from it too. Hope others enjoy it as much as I do. David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/ -- Josef Moravec josef.moravec@gmail.com <mailto:josef.moravec@gmail.com>
participants (3)
-
dcook＠prosentient.com.au -
Fridolin SOMERS -
Josef Moravec