[Koha-bugs] [Bug 21872] Elasticsearch indexing faster by making it multi-threaded

[bugzilla-daemon at bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21872

--- Comment #60 from David Cook <dcook at prosentient.com.au> ---
(In reply to Martin Renvoize from comment #57)
> Finally, I'll be looking into the possibility of race conditions being
> introduced with this.  We had to introduce lock files for the zebra indexer
> as overlapping runs of the script could cause problems, especially with the
> query that got the list of bib/auths to index during each run.  I'm vaguely
> feeling that might also be a problem here, but I'm not entirely sure yet as
> I'm still looking at how the iterator is being built.
> 

I am also concerned about there not being a lock file. I suppose I'm less
concerned about race conditions so much as accidentally running multiple
indexing runs before the first has even completed.

I was thinking about the scenario you mentioned where the parent process dies
and there's multiple child processes. I would be concerned that the lock would
be lost when the parent dies, although
https://perldoc.perl.org/functions/flock.html says that locks are inherited
across fork calls. In hindsight, I was thinking about the fork and exec
(http://www.wumpus-cave.net/2014/04/21/underappreciated-perl-passing-file-descriptors/),
but that shouldn't be an issue here.

So yeah... I think adding a lock file would be trivial but very worthwhile.

-- 
You are receiving this mail because:
You are watching all bug changes.