[Koha-bugs] [Bug 32594] Add a dedicated ES indexing background worker

[bugzilla-daemon at bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594

--- Comment #2 from David Cook <dcook at prosentient.com.au> ---
(In reply to Nick Clemens from comment #0)
> When pushing the ES indexing jobs to the background queue, we have found it
> is hard for the workers to keep up in production.
> 
> During high circulation times there are many single record reindexes
> produced.
> 
> It would be nice to have a dedicated ES worker that can batch these to
> reduce the processing time and reduce the hits to the ES server

I like the idea of batching together the single record indexes. 

I think I'd prefer if this worker created a new message that contained multiple
job_ids, but background_jobs_worker.pl only supports a single job_id. Maybe we
should enhance background_jobs_worker.pl to be able to take multiple job IDs...

I'm not super familiar with alarm, so I'm curious at what point in code it is
able to interrupt. It seems like it could introduce a race condition...

(If you received the frame, and then the alarm handler fired, the frame would
be acknowledged before it's processed. But the job should be handled in the
next alarm. I suppose you might double acknowledge the same frame by accident.
I don't know if that creates a warning or an error...)

--

Marking "Failed QA" as there's some unused copy/paste code that needs to be
removed (e.g. "process_job"). You'll also need to take into account Bug 32481.

I suspect that you might also run into memory problems over time since you're
not doing any forking. I reckon you could fork in the commit_records() function
to do the actual indexing, since that's the point that will be most subject to
spikes in memory usage.

-- 
You are receiving this mail because:
You are watching all bug changes.