[Koha-devel] Announcing an alternate indexing daemon for Zebra which is more memory efficient

[dcook at prosentient.com.au]

How are people going with this? I've been running it for about 1.5 months now with 0 complaints.

As I just noted on 27267, I just implemented it on a small cloud server with only 4GB RAM and I was able to save 500MB switching to the alternative indexer. Quite a big win in a resource constrained environment!

David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia

Office: 02 9212 0899
Online: 02 8005 0595

-----Original Message-----
From: Fridolin SOMERS <fridolin.somers at biblibre.com> 
Sent: Thursday, 7 January 2021 7:08 PM
To: dcook at prosentient.com.au; koha-devel at lists.koha-community.org
Subject: Re: [Koha-devel] Announcing an alternate indexing daemon for Zebra which is more memory efficient

Ah of course, rebuild_zebra.pl itself has lock system.
It ends if you dont use --wait-for-lock.

Thanks ;)

Le 07/01/2021 à 00:26, dcook at prosentient.com.au a écrit :
> There's no need for a couple reasons.
> 
> First, all child process operations for my "zebra_indexer.pl" (ie checking for records to index and the actual indexing) cause the parent indexing daemon to block (by using waitpid($pid,0)) so there should be no overlapping runs.
> 
> Second, even if multiple rebuild_zebra.pl processes were launched magically, the built-in locking in rebuild_zebra.pl would prevent any but the 1st process from running.
> 
> So it's all good 😃
> 
> David Cook
> Software Engineer
> Prosentient Systems
> Suite 7.03
> 6a Glen St
> Milsons Point NSW 2061
> Australia
> 
> Office: 02 9212 0899
> Online: 02 8005 0595
> 
> -----Original Message-----
> From: Koha-devel <koha-devel-bounces at lists.koha-community.org> On 
> Behalf Of Fridolin SOMERS
> Sent: Thursday, 7 January 2021 4:01 AM
> To: koha-devel at lists.koha-community.org
> Subject: Re: [Koha-devel] Announcing an alternate indexing daemon for 
> Zebra which is more memory efficient
> 
> Oooh nice.
> 
> Is there a protection against several launch ? Like in actual daemon --wait-for-lock ?
> Or is there no need ?
> 
> Regards,
> 
> Le 06/01/2021 à 08:50, Josef Moravec a écrit :
>> Hello David,
>>
>> thanks for sharing, it looks great! We are going to test it on one of 
>> our servers. We'll report how things are going probably in few days...
>>
>>
>> Josef
>>
>> On Wed, Jan 6, 2021 at 4:28 AM <dcook at prosentient.com.au 
>> <mailto:dcook at prosentient.com.au>> wrote:
>>
>>      Hi all,____
>>
>>      __ __
>>
>>      I’ve written a new indexing daemon for Zebra which you can find at
>>      https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27267
>>      <https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27267>.
>>      It’s dead easy to use. Once the daemon is installed, you just need
>>      to update /etc/default/koha-common and you’ll be using it.____
>>
>>      __ __
>>
>>      It only uses 6MB of RAM (instead of the 170-180MB I’ve seen
>>      rebuild_zebra.pl <http://rebuild_zebra.pl> use in daemon mode), so
>>      it scales well on servers that host many instances of Koha. (For
>>      example, if you have 50 Koha instances on a server, you’ll use 300MB
>>      instead of 9GB for idle Zebra indexing daemons.) That said, this
>>      indexing daemon does launch rebuild_zebra.pl
>>      <http://rebuild_zebra.pl> in non-daemon mode when necessary, so you
>>      still need to have memory available to do actual work. Making
>>      rebuild_zebra.pl <http://rebuild_zebra.pl> more memory efficient is
>>      another task in itself and not a pressing issue atm.____
>>
>>      __ __
>>
>>      I’m planning to use it shortly myself, but I figured that I would
>>      share it with everyone else, so that everyone else could benefit
>>      from it too. ____
>>
>>      __ __
>>
>>      Hope others enjoy it as much as I do.____
>>
>>      __ __
>>
>>      David Cook____
>>
>>      Software Engineer____
>>
>>      Prosentient Systems____
>>
>>      Suite 7.03____
>>
>>      6a Glen St____
>>
>>      Milsons Point NSW 2061____
>>
>>      Australia____
>>
>>      __ __
>>
>>      Office: 02 9212 0899____
>>
>>      Online: 02 8005 0595____
>>
>>      __ __
>>
>>      _______________________________________________
>>      Koha-devel mailing list
>>      Koha-devel at lists.koha-community.org
>>      <mailto:Koha-devel at lists.koha-community.org>
>>      https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
>>      <https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel>
>>      website : https://www.koha-community.org/
>>      <https://www.koha-community.org/>
>>      git : https://git.koha-community.org/ <https://git.koha-community.org/>
>>      bugs : https://bugs.koha-community.org/
>>      <https://bugs.koha-community.org/>
>>
>>
>>
>> --
>> Josef Moravec
>> josef.moravec at gmail.com <mailto:josef.moravec at gmail.com>
>>
>>
>> _______________________________________________
>> Koha-devel mailing list
>> Koha-devel at lists.koha-community.org
>> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
>> website : https://www.koha-community.org/ git :
>> https://git.koha-community.org/ bugs :
>> https://bugs.koha-community.org/
>>
> 
> --
> Fridolin SOMERS <fridolin.somers at biblibre.com> Software and system 
> maintainer 🦄
> BibLibre, France
> _______________________________________________
> Koha-devel mailing list
> Koha-devel at lists.koha-community.org
> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
> website : https://www.koha-community.org/ git : 
> https://git.koha-community.org/ bugs : 
> https://bugs.koha-community.org/
> 
> 

--
Fridolin SOMERS <fridolin.somers at biblibre.com> Software and system maintainer 🦄
BibLibre, France