[Bug 27344] New: Implement Elastic's update_index_background using Koha::BackgroundJob
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Bug ID: 27344 Summary: Implement Elastic's update_index_background using Koha::BackgroundJob Change sponsored?: --- Product: Koha Version: unspecified Hardware: All OS: All Status: ASSIGNED Severity: enhancement Priority: P5 - low Component: Architecture, internals, and plumbing Assignee: jonathan.druart@bugs.koha-community.org Reporter: jonathan.druart@bugs.koha-community.org QA Contact: testopia@bugs.koha-community.org Blocks: 27341 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27341 [Bug 27341] Performance 21.05 - Omnibus -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #1 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 114877 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=114877&action=edit Bug 27344: Implement Elastic's update_index_background using Koha::BackgroundJob This patch adds a background job submodule, UpdateElasticIndex, to deal with async ES index update (not the deletion). Using NYTProf (on a checkin): Without 618ms, executing 35676 statements and 26355 subroutine calls in 266 source files and 83 string evals. With 521ms, executing 13282 statements and 7979 subroutine calls in 195 source files and 26 string evals. However there are some problems with this patch: 1. We don't want *all* the index update to be in the background_jobs table (we could add a filter on the list view) 2. We don't track the "progress" of the job as we are sending all the records to Elastic. It is okish in my opinion but it must be noted. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |In Discussion CC| |nick@bywatersolutions.com --- Comment #2 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Nick, can you have a look at this patch please? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #3 from Nick Clemens <nick@bywatersolutions.com> --- In brief testing this works, in addition to noted issues: package Koha::BackgroundJob; sub enqueue { ... my $borrowernumber = C4::Context->userenv->{number}; # FIXME Handle non GUI calls if I run a batch modification the indexing fails when called from the background job -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Séverine Queune <severine.queune@bulac.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nicolas.legrand@bulac.fr, | |severine.queune@bulac.fr -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #4 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 116252 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=116252&action=edit Bug 27344: Set job.borrowernumber to undef if enqueued from cli script -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |martin.renvoize@ptfs-europe | |.com --- Comment #5 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- I feel like this should be handled in it's own indexing daemon more like how we do zebra personally.. it feels a bit superfluous to requirements to add it into the general queue. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #6 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to Martin Renvoize from comment #5)
I feel like this should be handled in it's own indexing daemon more like how we do zebra personally.. it feels a bit superfluous to requirements to add it into the general queue.
Why another daemon when we have a generic mechanism to deal such background jobs? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #7 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- (In reply to Jonathan Druart from comment #6)
(In reply to Martin Renvoize from comment #5)
I feel like this should be handled in it's own indexing daemon more like how we do zebra personally.. it feels a bit superfluous to requirements to add it into the general queue.
Why another daemon when we have a generic mechanism to deal such background jobs?
Or at least a distinct worked for just this type of Job... I don't think it would be OK for these jobs to get queued behind other, perhaps slow running, background jobs. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #8 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Thinking about this more.. I think we need to teach our worked daemon to accept a list of jobs it can work on.. that way we can utilise the background jobs processing whilst also allowing different workers to concentrate on different jobs. For this case, I think having a dedicated indexing worker would be the best option. I can work on that in another bug to not block this one. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #9 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- (In reply to Jonathan Druart from comment #1)
However there are some problems with this patch: 1. We don't want *all* the index update to be in the background_jobs table (we could add a filter on the list view)
Why not.. do we not have a cleanup process to clean down completed jobs after a period? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #10 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to Martin Renvoize from comment #8)
Thinking about this more.. I think we need to teach our worked daemon to accept a list of jobs it can work on.. that way we can utilise the background jobs processing whilst also allowing different workers to concentrate on different jobs. For this case, I think having a dedicated indexing worker would be the best option.
I can work on that in another bug to not block this one.
Yes, that's definitely something I had in mind when I wrote bug 22417, configuration for the different jobs. But I think by default it must be kept as simple as possible. Sysops who wants more sophisticate solutions will implement it (like if we need to modify the RabbitMQ config). Having an optional parameter to pass a list of "job type" is a good idea and could be implemented easily. (In reply to Martin Renvoize from comment #9)
(In reply to Jonathan Druart from comment #1)
However there are some problems with this patch: 1. We don't want *all* the index update to be in the background_jobs table (we could add a filter on the list view)
Why not.. do we not have a cleanup process to clean down completed jobs after a period?
Yes, that could be done. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #11 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- (In reply to Jonathan Druart from comment #1)
2. We don't track the "progress" of the job as we are sending all the records to Elastic. It is okish in my opinion but it must be noted.
I don't see any way of getting at status information for bulk requests in elastic.. I don't think it's hugely required to be able to report this back to the end user anyway. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au --- Comment #12 from David Cook <dcook@prosentient.com.au> --- (In reply to Martin Renvoize from comment #8)
For this case, I think having a dedicated indexing worker would be the best option.
10/10 agreed -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #13 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 116931 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=116931&action=edit Bug 27344: Add --job-type to background_jobs_worker.pl This will let sysop adjust the number of workers and how they want to manage them. For instance one could want to have one worker for ES indexation and another worker for other jobs, to prevent ES index to be stuck behind bigger batch process. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #14 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- David, Martin, here it is! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #15 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Hmm, I'm not contemplating whether we should actually be de-coupling 'jobs' from 'queues'.. or letting the 'type' option take a negation. Right now we just have the two jobs (three when this patch is added).. but if we see it getting real adoption we could end up with lots of different jobs.. it would become tedious listing all but one.. as such we might either want to say !update_elastic_index for one worker whilst update_elastic_index for another.. rather than having to list all remaining jobs on your non elastic worker. Other queuing systems appear to use 'queues' as a second distinction instead.. so we could have a 'default' queue that the non-elastic indexing tasks go onto and another for the elastic one.. that way we can differentiate into groups rather than having to list all types.. Thoughts? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #16 from David Cook <dcook@prosentient.com.au> --- (In reply to Martin Renvoize from comment #15)
Other queuing systems appear to use 'queues' as a second distinction instead.. so we could have a 'default' queue that the non-elastic indexing tasks go onto and another for the elastic one.. that way we can differentiate into groups rather than having to list all types..
Agreed. I think nominating queues rather than job types for workers makes more sense. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #17 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- I think --types is a good start.. however I think we aught to split this bug.. so we can discuss and work through queue improvements distinctly from this job.. Mind if I create a couple of bugs... one for --type (and I'll grab your patch and put it there if that's OK Jonathan).. and one for implementing queues. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #18 from David Cook <dcook@prosentient.com.au> --- (In reply to Martin Renvoize from comment #17)
I think --types is a good start.. however I think we aught to split this bug.. so we can discuss and work through queue improvements distinctly from this job..
Mind if I create a couple of bugs... one for --type (and I'll grab your patch and put it there if that's OK Jonathan).. and one for implementing queues.
I think that --type is likely to be unwieldy. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #19 from David Cook <dcook@prosentient.com.au> --- Actually, I just realized that background_jobs_worker.pl has a design flaw regarding memory consumption, which will become more obvious when doing elastic indexing. I'll open a new task for that. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Ere Maijala <ere.maijala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ere.maijala@helsinki.fi -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #20 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- (In reply to David Cook from comment #19)
Actually, I just realized that background_jobs_worker.pl has a design flaw regarding memory consumption, which will become more obvious when doing elastic indexing. I'll open a new task for that.
Care to expand on this? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=27782 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=27783 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #21 from David Cook <dcook@prosentient.com.au> --- (In reply to Martin Renvoize from comment #20)
(In reply to David Cook from comment #19)
Actually, I just realized that background_jobs_worker.pl has a design flaw regarding memory consumption, which will become more obvious when doing elastic indexing. I'll open a new task for that.
Care to expand on this?
Ooops. I meant to link the new issue. Check out Bug 27756. Basically, Perl will never return memory back to the OS. It will re-use memory once it's freed internally within the Perl program, but any memory allocated to a Perl process will stay allocated by the OS for the life of that process. Just see Bug 27756 for more information heh. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |victor@tuxayo.net -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #22 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Do we still need this or we abandon? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #23 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- I still think it's a good idea myself. One to pursue next cycle :) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Discussion |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #24 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Why isn't this moving forward? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Arthur Suzuki <arthur.suzuki@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |arthur.suzuki@biblibre.com Status|Needs Signoff |Patch doesn't apply --- Comment #25 from Arthur Suzuki <arthur.suzuki@biblibre.com> --- Patch doesn't apply on current master for me : <pre> arthur@h:~/repos/koha : $ git bz apply 27344 Bug 27344 - Implement Elastic's update_index_background using Koha::BackgroundJob 114877 - Bug 27344: Implement Elastic's update_index_background using Koha::BackgroundJob 116252 - Bug 27344: Set job.borrowernumber to undef if enqueued from cli script 116931 - Bug 27344: Add --job-type to background_jobs_worker.pl Apply? [(y)es, (n)o, (i)nteractive] y Application de Bug 27344: Implement Elastic's update_index_background using Koha::BackgroundJob Utilisation de l'information de l'index pour reconstruire un arbre de base... M Koha/BackgroundJob.pm M Koha/SearchEngine/Elasticsearch/Indexer.pm M koha-tmpl/intranet-tmpl/prog/en/modules/admin/background_jobs.tt M misc/background_jobs_worker.pl Retour à un patch de la base et fusion à 3 points... Fusion automatique de misc/background_jobs_worker.pl CONFLIT (contenu) : Conflit de fusion dans misc/background_jobs_worker.pl Fusion automatique de koha-tmpl/intranet-tmpl/prog/en/modules/admin/background_jobs.tt CONFLIT (contenu) : Conflit de fusion dans koha-tmpl/intranet-tmpl/prog/en/modules/admin/background_jobs.tt Fusion automatique de Koha/SearchEngine/Elasticsearch/Indexer.pm Fusion automatique de Koha/BackgroundJob.pm CONFLIT (contenu) : Conflit de fusion dans Koha/BackgroundJob.pm error: Échec d'intégration des modifications. l'application de la rustine a échoué à 0001 Bug 27344: Implement Elastic's update_index_background using Koha::BackgroundJob astuce: Utilisez 'git am --show-current-patch=diff' pour visualiser le patch en échec When you have resolved this problem run "git bz apply --continue". If you would prefer to skip this patch, instead run "git bz apply --skip". To restore the original branch and stop patching run "git bz apply --abort". Patch left in /tmp/Bug-27344-Implement-Elastics-updateindexbackground-HVa3WT.patch </pre> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #114877|0 |1 is obsolete| | Attachment #116252|0 |1 is obsolete| | Attachment #116931|0 |1 is obsolete| | --- Comment #26 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 132444 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132444&action=edit Bug 27344: Implement Elastic's update_index_background using Koha::BackgroundJob This patch adds a background job submodule, UpdateElasticIndex, to deal with async ES index update (not the deletion). Using NYTProf (on a checkin): Without 618ms, executing 35676 statements and 26355 subroutine calls in 266 source files and 83 string evals. With 521ms, executing 13282 statements and 7979 subroutine calls in 195 source files and 26 string evals. However there are some problems with this patch: 1. We don't want *all* the index update to be in the background_jobs table (we could add a filter on the list view) 2. We don't track the "progress" of the job as we are sending all the records to Elastic. It is okish in my opinion but it must be noted. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #27 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 132445 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132445&action=edit Bug 27344: Set job.borrowernumber to undef if enqueued from cli script -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #28 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 132446 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132446&action=edit Bug 27344: Add --job-type to background_jobs_worker.pl This will let sysop adjust the number of workers and how they want to manage them. For instance one could want to have one worker for ES indexation and another worker for other jobs, to prevent ES index to be stuck behind bigger batch process. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #29 from Julian Maurice <julian.maurice@biblibre.com> --- Created attachment 132586 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132586&action=edit Bug 27344: Replace --job-type by --job-queue This patch adds a new column background_jobs.queue, which default to 'default' By default, new jobs are enqueued into this default queue, and the background job worker will subscribe to the default queue unless told otherwise by the --job-queue option The new job UpdateElasticIndex is automatically enqueued in another queue named 'index' So you can have 'default' worker with misc/background_jobs_worker.pl and a dedicated indexing worker with misc/background_jobs_worker.pl --job-queue index This is to address bug 27344 comment #15 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #30 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 132609 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132609&action=edit Bug 27344: Add missing include file -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132444|0 |1 is obsolete| | --- Comment #31 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 132610 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132610&action=edit Bug 27344: Implement Elastic's update_index_background using Koha::BackgroundJob This patch adds a background job submodule, UpdateElasticIndex, to deal with async ES index update (not the deletion). Using NYTProf (on a checkin): Without 618ms, executing 35676 statements and 26355 subroutine calls in 266 source files and 83 string evals. With 521ms, executing 13282 statements and 7979 subroutine calls in 195 source files and 26 string evals. However there are some problems with this patch: 1. We don't want *all* the index update to be in the background_jobs tabtle (we could add a filter on the list view) 2. We don't track the "progress" of the job as we are sending all the records to Elastic. It is okish in my opinion but it must be noted. Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132445|0 |1 is obsolete| | --- Comment #32 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 132611 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132611&action=edit Bug 27344: Set job.borrowernumber to undef if enqueued from cli script Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132446|0 |1 is obsolete| | --- Comment #33 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 132612 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132612&action=edit Bug 27344: Add --job-type to background_jobs_worker.pl This will let sysop adjust the number of workers and how they want to manage them. For instance one could want to have one worker for ES indexation and another worker for other jobs, to prevent ES index to be stuck behind bigger batch process. Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132586|0 |1 is obsolete| | --- Comment #34 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 132613 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132613&action=edit Bug 27344: Replace --job-type by --job-queue This patch adds a new column background_jobs.queue, which default to 'default' By default, new jobs are enqueued into this default queue, and the background job worker will subscribe to the default queue unless told otherwise by the --job-queue option The new job UpdateElasticIndex is automatically enqueued in another queue named 'index' So you can have 'default' worker with misc/background_jobs_worker.pl and a dedicated indexing worker with misc/background_jobs_worker.pl --job-queue index This is to address bug 27344 comment #15 Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |27783 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 [Bug 27783] Add handling for 'Queues' in the background job queue -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #35 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Right.. I'm going to grab Julians final patch and move it to bug 27783 as I think it needs a little more thought/work. (Including adding a missing DB update that adds the new column and update existing jobs with their queue).. Then we can QA this piece in isolation. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132613|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Arthur Suzuki <arthur.suzuki@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132609|0 |1 is obsolete| | --- Comment #36 from Arthur Suzuki <arthur.suzuki@biblibre.com> --- Created attachment 132645 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132645&action=edit Bug 27344: Add missing include file Signed-off-by: Arthur Suzuki <arthur.suzuki@biblibre.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #37 from Arthur Suzuki <arthur.suzuki@biblibre.com> --- Created attachment 132646 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132646&action=edit Bug 27344: (QA follow-up) typo -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132610|0 |1 is obsolete| | --- Comment #38 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 132794 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132794&action=edit Bug 27344: Implement Elastic's update_index_background using Koha::BackgroundJob This patch adds a background job submodule, UpdateElasticIndex, to deal with async ES index update (not the deletion). Using NYTProf (on a checkin): Without 618ms, executing 35676 statements and 26355 subroutine calls in 266 source files and 83 string evals. With 521ms, executing 13282 statements and 7979 subroutine calls in 195 source files and 26 string evals. However there are some problems with this patch: 1. We don't want *all* the index update to be in the background_jobs tabtle (we could add a filter on the list view) 2. We don't track the "progress" of the job as we are sending all the records to Elastic. It is okish in my opinion but it must be noted. Signed-off-by: Arthur Suzuki <arthur.suzuki@biblibre.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132611|0 |1 is obsolete| | --- Comment #39 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 132795 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132795&action=edit Bug 27344: Set job.borrowernumber to undef if enqueued from cli script Signed-off-by: Arthur Suzuki <arthur.suzuki@biblibre.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132612|0 |1 is obsolete| | --- Comment #40 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 132796 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132796&action=edit Bug 27344: Add --job-type to background_jobs_worker.pl This will let sysop adjust the number of workers and how they want to manage them. For instance one could want to have one worker for ES indexation and another worker for other jobs, to prevent ES index to be stuck behind bigger batch process. Signed-off-by: Arthur Suzuki <arthur.suzuki@biblibre.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132645|0 |1 is obsolete| | --- Comment #41 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 132797 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132797&action=edit Bug 27344: Add missing include file Signed-off-by: Arthur Suzuki <arthur.suzuki@biblibre.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132646|0 |1 is obsolete| | --- Comment #42 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 132798 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132798&action=edit Bug 27344: (QA follow-up) typo Signed-off-by: Arthur Suzuki <arthur.suzuki@biblibre.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Arthur Suzuki <arthur.suzuki@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |30345 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30345 [Bug 30345] Koha::BackgroundJob->enqueue should set borrowernumber=undef if no userenv -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |tomascohen@gmail.com See Also|https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=27783 | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks|27783 | Depends on| |27783 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 [Bug 27783] Add handling for 'Queues' in the background job queue -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132794|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132795|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132796|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132797|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132798|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #43 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 132843 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132843&action=edit Bug 27344: Implement Elastic's update_index_background using Koha::BackgroundJob This patch adds a background job submodule, UpdateElasticIndex, to deal with async ES index update (not the deletion). Using NYTProf (on a checkin): Without 618ms, executing 35676 statements and 26355 subroutine calls in 266 source files and 83 string evals. With 521ms, executing 13282 statements and 7979 subroutine calls in 195 source files and 26 string evals. However there are some problems with this patch: 1. We don't want *all* the index update to be in the background_jobs tabtle (we could add a filter on the list view) 2. We don't track the "progress" of the job as we are sending all the records to Elastic. It is okish in my opinion but it must be noted. Signed-off-by: Arthur Suzuki <arthur.suzuki@biblibre.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #44 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 132844 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132844&action=edit Bug 27344: Add missing include file Signed-off-by: Arthur Suzuki <arthur.suzuki@biblibre.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #45 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 132845 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132845&action=edit Bug 27344: Adapt to bug 30181 changes Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on|30345 | Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30345 [Bug 30345] Koha::BackgroundJob->enqueue should set borrowernumber=undef if no userenv -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |30360 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30360 [Bug 30360] Add helper methods to Koha::BackgroundJobs -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #46 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- This got broken at some point data: {"messages":[{"code":"index_error","type":"error","error":"Can't locate object method \"new\" via package \"Koha::SearchEngine::Indexer\" (perhaps you forgot to load \"Koha::SearchEngine::Indexer\"?) at /kohadevbox/koha/Koha/BackgroundJob/UpdateElasticIndex.pm line 78.\n"}],"record_ids":["14"],"report":{"total_records":1,"total_success":0},"record_server":"biblioserver"} -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #47 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 132969 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132969&action=edit Bug 27344: Add missing imports -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #48 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 132970 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132970&action=edit Bug 27344: Warn the error in the log Otherwise info about the failure is hidden in DB -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #49 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Minor: Progress is 0/1 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #50 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Batch item modification (for instance) will run in background, but will also run a new task for the indexation. I don't think we should run another task for the indexation in case of batch operations. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #51 from Tomás Cohen Arazi <tomascohen@gmail.com> --- (In reply to Jonathan Druart from comment #50)
Batch item modification (for instance) will run in background, but will also run a new task for the indexation. I don't think we should run another task for the indexation in case of batch operations.
This is your patch, right? We just moved the job_type/queue stuff to another bug. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |30459 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30459 [Bug 30459] BatchDeleteAuthority task does not deal with indexation correctly -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |30460 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30460 [Bug 30460] BatchDeleteBiblio task does not deal with indexation correctly -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |30462 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30462 [Bug 30462] Should the background job list view hide index tasks by default? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |30463 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30463 [Bug 30463] Authority records are indexed on the fly -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |30465 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30465 [Bug 30465] BatchUpdateBiblio task does not deal with indexation correctly -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA --- Comment #52 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- I have opened several bug reports. There is a lot of work to provide before considering this one ready. I am not sure about the relations between the bug reports, but I think at least bug 30465 is a blocking this one. Otherwise we will see 1 task + X indexation tasks when you update X biblio records using the batch record modification tools. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |30465 Depends on|30465 | Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30465 [Bug 30465] BatchUpdateBiblio task does not deal with indexation correctly -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |30467 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30467 [Bug 30467] BatchDeleteItem task does not deal with indexation correctly -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #53 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 132994 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132994&action=edit Bug 27344: Fix tests -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #54 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 132995 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132995&action=edit Bug 27344: Fix call to update_index with invalid ids Can't use an undefined value as an ARRAY reference at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Indexer.pm line 121. Caught that when working on follow-up bug reports. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff --- Comment #55 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Ready for testing again. I think we should push all the related bug reports along with this one. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #56 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 133033 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=133033&action=edit Bug 27344: Add error detail if something wrong happened -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #57 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- I've pushed a remote branch with the different dependencies applied https://gitlab.com/joubu/Koha/-/commits/bug_27344 If we decide to enqueue a new task for the indexing step (for the other background jobs), we just need to remove the { index_now => 1 } flag we pass to ->index_records. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Bug 27344 depends on bug 27783, which changed state. Bug 27783 Summary: Introduce background job queues https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783 What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to master |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |30654 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30654 [Bug 30654] Even with RabbitMQ enabled, we should poll the database for jobs at worker startup -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks|30654 | Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30654 [Bug 30654] Even with RabbitMQ enabled, we should poll the database for jobs at worker startup -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132994|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132995|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132969|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132970|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132843|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132844|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132845|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #133033|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #58 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 134406 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=134406&action=edit Bug 27344: Implement Elastic's update_index_background using Koha::BackgroundJob This patch adds a background job submodule, UpdateElasticIndex, to deal with async ES index update (not the deletion). Using NYTProf (on a checkin): Without 618ms, executing 35676 statements and 26355 subroutine calls in 266 source files and 83 string evals. With 521ms, executing 13282 statements and 7979 subroutine calls in 195 source files and 26 string evals. However there are some problems with this patch: 1. We don't want *all* the index update to be in the background_jobs tabtle (we could add a filter on the list view) 2. We don't track the "progress" of the job as we are sending all the records to Elastic. It is okish in my opinion but it must be noted. Signed-off-by: Arthur Suzuki <arthur.suzuki@biblibre.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #59 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 134407 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=134407&action=edit Bug 27344: Add missing include file Signed-off-by: Arthur Suzuki <arthur.suzuki@biblibre.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #60 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 134408 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=134408&action=edit Bug 27344: Adapt to bug 30181 changes Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #61 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 134409 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=134409&action=edit Bug 27344: Add missing imports -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #62 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 134410 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=134410&action=edit Bug 27344: Warn the error in the log Otherwise info about the failure is hidden in DB -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #63 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 134411 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=134411&action=edit Bug 27344: Fix tests -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #64 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 134412 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=134412&action=edit Bug 27344: Fix call to update_index with invalid ids Can't use an undefined value as an ARRAY reference at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Indexer.pm line 121. Caught that when working on follow-up bug reports. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #65 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 134413 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=134413&action=edit Bug 27344: Add error detail if something wrong happened -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |30464 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30464 [Bug 30464] BatchUpdateAuthority task does not deal with indexation correctly -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |rel_22_05_candidate -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|--- |Medium patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |tomascohen@gmail.com |y.org | Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |22.05.00 released in| | Status|Passed QA |Pushed to master -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 --- Comment #66 from Fridolin Somers <fridolin.somers@biblibre.com> --- Pushed to master for 22.05, thanks to everybody involved 🦄 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |30692 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30692 [Bug 30692] Wrong progress displayed for ES indexing tasks -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks|30692 | Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30692 [Bug 30692] Wrong progress displayed for ES indexing tasks -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|rel_22_05_candidate |release-notes-needed -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks|27341 | Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27341 [Bug 27341] Performance 21.05 - Omnibus -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |30710 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30710 [Bug 30710] Background tasks can be called multiple times on batch item deletion -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks|30710 | Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30710 [Bug 30710] Background tasks can be called multiple times on batch item deletion -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |kyle@bywatersolutions.com Status|Pushed to master |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |30822 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30822 [Bug 30822] BatchCommit does not deal with indexation correctly -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |30954 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30954 [Bug 30954] includes/background_jobs_update_elastic_index.inc must be removed -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |31010 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31010 [Bug 31010] BatchRevert does not deal with indexation correctly -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|release-notes-needed | -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org