[Koha-bugs] [Bug 21872] Elasticsearch indexing faster by making it multi-threaded

[bugzilla-daemon at bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21872

Martin Renvoize <martin.renvoize at ptfs-europe.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|Signed Off                  |Failed QA

--- Comment #57 from Martin Renvoize <martin.renvoize at ptfs-europe.com> ---
QA looking here.

I've got a couple of points to make before continuing.

1) The 'die' on fork failure isn't cleaning up after itself.. imagine a case
where we want to sporn 5 subprocesses, it we get to process 4 and then run out
of memory for example. The parent script will die and leave behind zombie child
processes.
2) It doesn't look like there's any form of signal handling here and as such a
CTRL+C for example could end up leaving zombie processes too.

I'm also wrapping my head around the use of wait vs waitpid here.. I remember
tripping myself up using them before, but can't remember the details well
enough right now to be confident I've not missed something.

Finally, I'll be looking into the possibility of race conditions being
introduced with this.  We had to introduce lock files for the zebra indexer as
overlapping runs of the script could cause problems, especially with the query
that got the list of bib/auths to index during each run.  I'm vaguely feeling
that might also be a problem here, but I'm not entirely sure yet as I'm still
looking at how the iterator is being built.

It's great to see this work however.. I'd love to see if make it into the 19.05
release.

Failing for the first issue raised above for now.

(I found https://www.perl.com/article/fork-yeah-/ pretty helpful whilst QAing
this.. it gave me the insight to spot the above issues where I may have missed
them otherwise)

-- 
You are receiving this mail because:
You are watching all bug changes.