[Bug 32594] New: Add a dedicated ES indexing background worker
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Bug ID: 32594 Summary: Add a dedicated ES indexing background worker Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Searching - Elasticsearch Assignee: koha-bugs@lists.koha-community.org Reporter: nick@bywatersolutions.com When pushing the ES indexing jobs to the background queue, we have found it is hard for the workers to keep up in production. During high circulation times there are many single record reindexes produced. It would be nice to have a dedicated ES worker that can batch these to reduce the processing time and reduce the hits to the ES server -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #1 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 145145 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145145&action=edit Bug 32594: Add a dedicated Elasticsearch biblio indexing background queue Currently we generate large numebrs if single record reindex for circulation and other actions. It can take a long time to process these as we need to load the ES settings for each. This patch updates the Elasticsearch background jobs to throw records into a new queue that can be processed by it's own worker and adds a dedicated worker that batches the jobs every 1 second. To test: 1 - Apply patches, set SearchEngine system preference to 'Elasticsearch' 2 - perl misc/search_tools/es_indexer_daemon.pl 3 - Leave the running in terminal and perform actions in staff interface: - Checking out a bib - Returning a bib - Editing a single bib - Editing a single item - Batch editing bibs - Batch editing items 4 - Confirm for each action that records are updated in search/search results 5 - Stop the script 6 - set SearchEngine system preference to 'Zebra' 7 - perl misc/search_tools/es_indexer_daemon.pl 8 - Script dies as Elasticsearch not enabled -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |nick@bywatersolutions.com |ity.org | Blocks| |32572 CC| |dcook@prosentient.com.au, | |jonathan.druart+koha@gmail. | |com Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32572 [Bug 32572] We should have a background queue for each job type -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=32481 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #2 from David Cook <dcook@prosentient.com.au> --- (In reply to Nick Clemens from comment #0)
When pushing the ES indexing jobs to the background queue, we have found it is hard for the workers to keep up in production.
During high circulation times there are many single record reindexes produced.
It would be nice to have a dedicated ES worker that can batch these to reduce the processing time and reduce the hits to the ES server
I like the idea of batching together the single record indexes. I think I'd prefer if this worker created a new message that contained multiple job_ids, but background_jobs_worker.pl only supports a single job_id. Maybe we should enhance background_jobs_worker.pl to be able to take multiple job IDs... I'm not super familiar with alarm, so I'm curious at what point in code it is able to interrupt. It seems like it could introduce a race condition... (If you received the frame, and then the alarm handler fired, the frame would be acknowledged before it's processed. But the job should be handled in the next alarm. I suppose you might double acknowledge the same frame by accident. I don't know if that creates a warning or an error...) -- Marking "Failed QA" as there's some unused copy/paste code that needs to be removed (e.g. "process_job"). You'll also need to take into account Bug 32481. I suspect that you might also run into memory problems over time since you're not doing any forking. I reckon you could fork in the commit_records() function to do the actual indexing, since that's the point that will be most subject to spikes in memory usage. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #3 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- I don't think this is the correct way. First I think we should have an entry in koha-conf to configure the workers, depending on their type. Here, for update_elastic_index jobs, we actually want to batch consume and process them all every X seconds (or we could decide every X jobs). I am trying something. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #4 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- On the other hand... we are having the Koha module loaded already, and it cannot be more efficient than that. I wanted to avoid a different script for a specific job type, but actually I am inclined to agree with the approach. My first idea was to have a Koha::BackgroundJobs::UpdateElasticIndex (notice the *s*) to process several jobs. We would send the job ids from an array constructed from the worker, with something in the config like { background_jobs: { update_elastic_index: { batch: 10 } } } to process per batch of 10. But it makes things more complicated and we don't want that for now. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |ASSIGNED Assignee|nick@bywatersolutions.com |jonathan.druart+koha@gmail. | |com Depends on| |32393, 32481 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32393 [Bug 32393] background job worker explodes if JSON is incorrect https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32481 [Bug 32481] Rabbit times out when too many jobs are queued and the response takes too long -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #5 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 145195 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145195&action=edit Bug 32594: Improve the ES daemon indexer -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #145195|0 |1 is obsolete| | --- Comment #6 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 145196 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145196&action=edit Bug 32594: Improve the ES daemon indexer -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #7 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Adding my bit to the discussion. We are not ready however: - I think we should deal with authorities here - We need to make the constant configurable (options of the script enough?) - Koha::Logger is logging to the opac log (this is true for bug 32393 as well). -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |In Discussion Assignee|jonathan.druart+koha@gmail. |nick@bywatersolutions.com |com | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #145196|0 |1 is obsolete| | --- Comment #8 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 145197 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145197&action=edit Bug 32594: Improve the ES daemon indexer -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #9 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- (In reply to Jonathan Druart from comment #7)
Adding my bit to the discussion. We are not ready however:
- I think we should deal with authorities here - We need to make the constant configurable (options of the script enough?) - Koha::Logger is logging to the opac log (this is true for bug 32393 as well).
Also I think it is confusing to have misc/search_tools/es_indexer_daemon.pl and misc/background_jobs_worker.pl -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|enhancement |critical --- Comment #10 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- We need this ASAP for 22.11. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |m.de.rooy@rijksmuseum.nl, | |martin.renvoize@ptfs-europe | |.com, tomascohen@gmail.com --- Comment #11 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- No-one else here? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #12 from David Cook <dcook@prosentient.com.au> --- Comment on attachment 145197 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145197 Bug 32594: Improve the ES daemon indexer Review of attachment 145197: --> (https://bugs.koha-community.org/bugzilla3/page.cgi?id=splinter.html&bug=32594&attachment=145197) ----------------------------------------------------------------- ::: misc/search_tools/es_indexer_daemon.pl @@ +126,4 @@
+ my @records; + for my $job ( @jobs ) { + my $args = $job->json->decode($job->data);
I suppose this line should have a try/catch since we're doing that for the other job handling code. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #13 from David Cook <dcook@prosentient.com.au> --- I don't use Elasticsearch with Koha yet, so I'm probably not the best positioned to comment. (In reply to Jonathan Druart from comment #9)
Also I think it is confusing to have misc/search_tools/es_indexer_daemon.pl and misc/background_jobs_worker.pl
I don't think there's an issue having a separate worker program, but "search_tools" probably isn't the right place for it. Perhaps we should have something like "misc/workers/background_jobs_worker.pl" and "misc/workers/es_indexer_daemon.pl" -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Discussion |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #145145|0 |1 is obsolete| | Attachment #145197|0 |1 is obsolete| | --- Comment #14 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 145704 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145704&action=edit Bug 32594: Add a dedicated Elasticsearch biblio indexing background queue Currently we generate large numebrs if single record reindex for circulation and other actions. It can take a long time to process these as we need to load the ES settings for each. This patch updates the Elasticsearch background jobs to throw records into a new queue that can be processed by it's own worker and adds a dedicated worker that batches the jobs every 1 second. To test: 1 - Apply patches, set SearchEngine system preference to 'Elasticsearch' 2 - perl misc/search_tools/es_indexer_daemon.pl 3 - Leave the running in terminal and perform actions in staff interface: - Checking out a bib - Returning a bib - Editing a single bib - Editing a single item - Batch editing bibs - Batch editing items 4 - Confirm for each action that records are updated in search/search results 5 - Stop the script 6 - set SearchEngine system preference to 'Zebra' 7 - perl misc/search_tools/es_indexer_daemon.pl 8 - Script dies as Elasticsearch not enabled -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #145704|0 |1 is obsolete| | --- Comment #15 from David Nind <david@davidnind.com> --- Created attachment 145711 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145711&action=edit Bug 32594: Add a dedicated Elasticsearch biblio indexing background queue Currently we generate large numebrs if single record reindex for circulation and other actions. It can take a long time to process these as we need to load the ES settings for each. This patch updates the Elasticsearch background jobs to throw records into a new queue that can be processed by it's own worker and adds a dedicated worker that batches the jobs every 1 second. To test: 1 - Apply patches, set SearchEngine system preference to 'Elasticsearch' 2 - perl misc/search_tools/es_indexer_daemon.pl 3 - Leave the running in terminal and perform actions in staff interface: - Checking out a bib - Returning a bib - Editing a single bib - Editing a single item - Batch editing bibs - Batch editing items 4 - Confirm for each action that records are updated in search/search results 5 - Stop the script 6 - set SearchEngine system preference to 'Zebra' 7 - perl misc/search_tools/es_indexer_daemon.pl 8 - Script dies as Elasticsearch not enabled Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david@davidnind.com --- Comment #16 from David Nind <david@davidnind.com> --- Testing notes (using KTD): 1. The path to the script for steps 2 and 7 is: perl misc/workers/es_indexer_daemon.pl -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Emily Lamancusa <emily.lamancusa@montgomerycountymd.gov> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |emily.lamancusa@montgomeryc | |ountymd.gov -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Patch doesn't apply --- Comment #17 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- error: sha1 information is lacking or useless (misc/background_jobs_worker.pl). -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact| |jonathan.druart+koha@gmail. | |com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #18 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Shouldn't we move misc/background_jobs_worker.pl to misc/workers/ as well? On another bug? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #19 from David Cook <dcook@prosentient.com.au> --- (In reply to Jonathan Druart from comment #18)
Shouldn't we move misc/background_jobs_worker.pl to misc/workers/ as well? On another bug?
+1 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Emily Lamancusa <emily.lamancusa@montgomerycountymd.gov> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Failed QA --- Comment #20 from Emily Lamancusa <emily.lamancusa@montgomerycountymd.gov> --- I encountered an error while testing this patch with batch item updates, in which one bad bib blocked the ES indexing update for every item in the batch. To reproduce the error: 1. Apply patches and run ktd --es7 up 2. Set SearchEngine system preference to 'Elasticsearch' 3. perl misc/search_tools/es_indexer_daemon.pl (leave running in the terminal while performing subsequent steps) 4. Perform an item search for all items where Collection is Reference 5. Export the results to a barcode file (898 items) 6. Upload this file to the Batch Item Modification Tool 7. Make an edit to the Full Call Number* and click Save 8. Notice that es_indexer_daemon prints an error to the console 9. Perform a catalog search 10. Limit the results with the Collection - Reference facet 11. Note that none of the results display the updated Call Number value *Or, as far as I can tell, any edit that will produce a change in the record for biblionumber 369 Error message printed to the console: ^ at /kohadevbox/koha/Koha/Biblio/Metadata.pm line 114. DEBUG - Update of elastic index failed with: Invalid data, cannot decode metadata object (biblio_metadata.id=368, biblionumber=369, format=marcxml, schema=MARC21, decoding_error=':8: parser error : PCDATA invalid Char value 31 <controlfield tag="001">00aD000015937</controlfield> ^ :9: parser error : PCDATA invalid Char value 31 <controlfield tag="004">00satmrnu0</controlfield> ^ :9: parser error : PCDATA invalid Char value 31 <controlfield tag="004">00satmrnu0</controlfield> ^ :9: parser error : PCDATA invalid Char value 31 <controlfield tag="004">00satmrnu0</controlfield> ^ :9: parser error : PCDATA invalid Char value 31 <controlfield tag="004">00satmrnu0</controlfield> ^ :10: parser error : PCDATA invalid Char value 31 <controlfield tag="008">00ar19881981bdkldan</controlfield> ^ :10: parser error : PCDATA invalid Char value 31 <controlfield tag="008">00ar19881981bdkldan</controlfield> ^ :10: parser error : PCDATA invalid Char value 31 <controlfield tag="008">00ar19881981bdkldan</controlfield> ^') -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff --- Comment #21 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- (In reply to Emily Lamancusa from comment #20)
I encountered an error while testing this patch with batch item updates, in which one bad bib blocked the ES indexing update for every item in the batch.
Hello Emily, I confirm the problem, but that's not directly linked with this patch. I am pretty sure this issue exists without this patch. When you create a new batch, 2 new jobs are created, the first one is modifying the items in database, the second one is updating elastic's index. If you look at the job list you will see that the second one has in error (still not clear, the status is "finished", but there is an error in the detail: " / records have successfully been reindexed. Some errors occurred.") That's not ideal but we should deal with that on a separate bug report in my opinion. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #22 from Emily Lamancusa <emily.lamancusa@montgomerycountymd.gov> --- (In reply to Jonathan Druart from comment #21)
(In reply to Emily Lamancusa from comment #20)
I encountered an error while testing this patch with batch item updates, in which one bad bib blocked the ES indexing update for every item in the batch.
Hello Emily, I confirm the problem, but that's not directly linked with this patch. I am pretty sure this issue exists without this patch.
When you create a new batch, 2 new jobs are created, the first one is modifying the items in database, the second one is updating elastic's index. If you look at the job list you will see that the second one has in error (still not clear, the status is "finished", but there is an error in the detail: " / records have successfully been reindexed. Some errors occurred.")
That's not ideal but we should deal with that on a separate bug report in my opinion.
Good to know - I didn't think to check whether the error was present without the patch (though it seems obvious in retrospect). Everything worked as intended on edits that didn't touch the broken bib, so I'll sign off here and open a new bug. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Emily Lamancusa <emily.lamancusa@montgomerycountymd.gov> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #145711|0 |1 is obsolete| | --- Comment #23 from Emily Lamancusa <emily.lamancusa@montgomerycountymd.gov> --- Created attachment 146402 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=146402&action=edit Bug 32594: Add a dedicated Elasticsearch biblio indexing background queue Currently we generate large numebrs if single record reindex for circulation and other actions. It can take a long time to process these as we need to load the ES settings for each. This patch updates the Elasticsearch background jobs to throw records into a new queue that can be processed by it's own worker and adds a dedicated worker that batches the jobs every 1 second. To test: 1 - Apply patches, set SearchEngine system preference to 'Elasticsearch' 2 - perl misc/search_tools/es_indexer_daemon.pl 3 - Leave the running in terminal and perform actions in staff interface: - Checking out a bib - Returning a bib - Editing a single bib - Editing a single item - Batch editing bibs - Batch editing items 4 - Confirm for each action that records are updated in search/search results 5 - Stop the script 6 - set SearchEngine system preference to 'Zebra' 7 - perl misc/search_tools/es_indexer_daemon.pl 8 - Script dies as Elasticsearch not enabled Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Emily Lamancusa <emily.lamancusa@montgomerycountymd.gov> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Emily Lamancusa <emily.lamancusa@montgomerycountymd.gov> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #24 from Emily Lamancusa <emily.lamancusa@montgomerycountymd.gov> --- Testing notes (using KTD): The path to the script for steps 2 and 7 is: perl misc/workers/es_indexer_daemon.pl -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #25 from Emily Lamancusa <emily.lamancusa@montgomerycountymd.gov> --- Bug 32920 created for the error noted in comment #20 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=32920 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Bug 32594 depends on bug 32393, which changed state. Bug 32393 Summary: background job worker explodes if JSON is incorrect https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32393 What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to oldstable |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #26 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 146831 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=146831&action=edit Bug 32594: (follow-up) Adjust logging per bug 32612 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |32992 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32992 [Bug 32992] Move background worker script to misc/workers -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #146402|0 |1 is obsolete| | Attachment #146831|0 |1 is obsolete| | --- Comment #27 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 147141 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=147141&action=edit Bug 32594: Add a dedicated Elasticsearch biblio indexing background queue Currently we generate large numebrs if single record reindex for circulation and other actions. It can take a long time to process these as we need to load the ES settings for each. This patch updates the Elasticsearch background jobs to throw records into a new queue that can be processed by it's own worker and adds a dedicated worker that batches the jobs every 1 second. To test: 1 - Apply patches, set SearchEngine system preference to 'Elasticsearch' 2 - perl misc/search_tools/es_indexer_daemon.pl 3 - Leave the running in terminal and perform actions in staff interface: - Checking out a bib - Returning a bib - Editing a single bib - Editing a single item - Batch editing bibs - Batch editing items 4 - Confirm for each action that records are updated in search/search results 5 - Stop the script 6 - set SearchEngine system preference to 'Zebra' 7 - perl misc/search_tools/es_indexer_daemon.pl 8 - Script dies as Elasticsearch not enabled Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Emily Lamancusa <emily.lamancusa@montgomerycountymd.gov> Bug 32594: (follow-up) Adjust logging per bug 32612 JD amended patch: tidy! There were tabs here... Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA --- Comment #28 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Not sure where my patches disappear, but that's not important. Ready to go! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |23.05.00 released in| | Status|Passed QA |Pushed to master -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #29 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Pushed to master for 23.05. Nice work everyone, thanks! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #30 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 147460 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=147460&action=edit Bug 32594: (QA follow-up) Adjust tests Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |additional_work_needed --- Comment #31 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- I think we forget an important part here, there is no daemon around the new script, and so it's not running (and records are not indexed!) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #32 from Tomás Cohen Arazi <tomascohen@gmail.com> --- (In reply to Jonathan Druart from comment #31)
I think we forget an important part here, there is no daemon around the new script, and so it's not running (and records are not indexed!)
Maybe bake it into koha-indexer? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|23.05.00 |23.05.00,22.11.04 released in| | Status|Pushed to master |Pushed to stable -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #33 from Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com> --- Nice work everyone! Pushed to stable for 22.11.x -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |33108 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33108 [Bug 33108] We need a way to launch the ES indexer automatically -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #34 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 147562 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=147562&action=edit Bug 32594: (QA follow-up) Fix POD Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |rel_22_11_candidate --- Comment #35 from Tomás Cohen Arazi <tomascohen@gmail.com> --- POD fix pushed to master. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #36 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Nick, I am seeing " / records have successfully been reindexed. Some errors occurred. " when showing a job detail. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #37 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Should we remove Koha::BackgroundJob::UpdateElasticIndex->progress? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |matt.blenkinsop@ptfs-europe | |.com Version(s)|23.05.00,22.11.04 |23.05.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #38 from Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com> --- Backport to 22.11.04 reversed while additional work is needed -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #39 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 148350 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=148350&action=edit Bug 32594: POC -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #40 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 148513 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=148513&action=edit Bug 32594: Mark jobs as started and finished -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #41 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- (In reply to Nick Clemens from comment #39)
Created attachment 148350 [details] [review] Bug 32594: POC
I don't think we should loop over the jobs, we are adding additional (almost useless) processing. Here we need to do things fast, so let's update them all at once. To prevent other scripts/jobs to use this, we should not provide a method (that does not feel correct). Now we need to decide if we allow indexing jobs from another worker. The detail page of the job is still wrong " / records have successfully been reindexed. Some errors occurred.". This is because it expects Koha::BackgroundJob::UpdateElasticIndex->process to have processed the job, and provided the correct info. IMO we should not provide 2 ways to process the jobs, only the new worker should do it. So let's remove the process method and simplify the job detail view. Sounds like we should move that to a separate bug... -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=33316 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #148513|0 |1 is obsolete| | --- Comment #42 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 148571 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=148571&action=edit Bug 32594: Mark jobs as started and finished Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #43 from Nick Clemens <nick@bywatersolutions.com> --- (In reply to Jonathan Druart from comment #41)
(In reply to Nick Clemens from comment #39)
Created attachment 148350 [details] [review] [review] Bug 32594: POC
I don't think we should loop over the jobs, we are adding additional (almost useless) processing. Here we need to do things fast, so let's update them all at once. To prevent other scripts/jobs to use this, we should not provide a method (that does not feel correct).
I was trying to set the number processed to the numebr in the job, I forgot we made all a size of 1 I squashed and mark yours SO
Now we need to decide if we allow indexing jobs from another worker. The detail page of the job is still wrong " / records have successfully been reindexed. Some errors occurred.".
Bug 33319 filed to improve As for processing:
IMO we should not provide 2 ways to process the jobs, only the new worker should do it. So let's remove the process method and simplify the job detail view. Sounds like we should move that to a separate bug...
I think that's a third job - we could have some plugins or some outside system wanting to handle indexation for some reason? But I am okay to remove too -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #148350|0 |1 is obsolete| | Attachment #148571|0 |1 is obsolete| | --- Comment #44 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 148588 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=148588&action=edit Bug 32594: Mark jobs as started and finished Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Restored authorship so that it shows the patch has been signed off. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to stable |ASSIGNED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to master -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|additional_work_needed | --- Comment #45 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Follow-up in master. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |33486 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33486 [Bug 33486] Remove Koha::BackgroundJob::UpdateElasticIndex->process routine -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|23.05.00 |23.05.00,22.11.06 released in| | Status|Pushed to master |Pushed to stable -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 --- Comment #46 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Many hands makes light work, thankyou everyone! Pushed to 22.11.x for the next release -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Bug 32594 depends on bug 32992, which changed state. Bug 32992 Summary: Move background worker script to misc/workers https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32992 What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to stable |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to stable |RESOLVED CC| |lucas@bywatersolutions.com Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |35086 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35086 [Bug 35086] Koha::SearchEngine::Elasticsearch::Indexer->update_index needs to commit in batches -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |36009 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36009 [Bug 36009] Document koha-worker --queue elastic_index -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |37787 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37787 [Bug 37787] Undocument koha-worker --queue elastic_index -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32594 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |39694 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39694 [Bug 39694] `es_indexer_daemon.pl` doesn't use batch_size in DB poll mode -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org