[Bug 28833] New: Speed up holds queue builder via parallel processing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Bug ID: 28833 Summary: Speed up holds queue builder via parallel processing Change sponsored?: --- Product: Koha Version: unspecified Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Hold requests Assignee: koha-bugs@lists.koha-community.org Reporter: kyle@bywatersolutions.com QA Contact: testopia@bugs.koha-community.org CC: gmcharlt@gmail.com The holds queue builder can take a very long time to run on systems with many holds. For example, a partner with 124,784 unfilled ( not found ) holds, is taking about 64 minutes to run. If we run that same number of holds in 5 parallel chunks ( splitting the number of records as evenly as possible, but *not* taking into account the holds per bib ), it takes 21.5 minutes. If we use 10 loops, it takes less then 14 minutes. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |kyle@bywatersolutions.com |ity.org | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 --- Comment #1 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 123656 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123656&action=edit Bug 28833: Speed up holds queue builder via parallel processing The holds queue builder can take a very long time to run on systems with many holds. For example, a partner with 124,784 unfilled ( not found ) holds, is taking about 64 minutes to run. If we run that same number of holds in 5 parallel chunks ( splitting the number of records as evenly as possible, but *not* taking into account the holds per bib ), it takes 21.5 minutes. If we use 10 loops, it takes less then 14 minutes. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 --- Comment #2 from Kyle M Hall <kyle@bywatersolutions.com> --- This patch provides a tremendous speedup for large numbers of holds. However, it causes HoldsQueue.t to fail spectacularly. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #123656|0 |1 is obsolete| | --- Comment #3 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 123657 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123657&action=edit Bug 28833: Speed up holds queue builder via parallel processing The holds queue builder can take a very long time to run on systems with many holds. For example, a partner with 124,784 unfilled ( not found ) holds, is taking about 64 minutes to run. If we run that same number of holds in 5 parallel chunks ( splitting the number of records as evenly as possible, but *not* taking into account the holds per bib ), it takes 21.5 minutes. If we use 10 loops, it takes less then 14 minutes. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #123657|0 |1 is obsolete| | --- Comment #4 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 123659 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123659&action=edit Bug 28833: Speed up holds queue builder via parallel processing The holds queue builder can take a very long time to run on systems with many holds. For example, a partner with 124,784 unfilled ( not found ) holds, is taking about 64 minutes to run. If we run that same number of holds in 5 parallel chunks ( splitting the number of records as evenly as possible, but *not* taking into account the holds per bib ), it takes 21.5 minutes. If we use 10 loops, it takes less then 14 minutes. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Jessie Zairo <jzairo@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jzairo@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #123659|0 |1 is obsolete| | --- Comment #5 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 123671 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123671&action=edit Bug 28833: Speed up holds queue builder via parallel processing The holds queue builder can take a very long time to run on systems with many holds. For example, a partner with 124,784 unfilled ( not found ) holds, is taking about 64 minutes to run. If we run that same number of holds in 5 parallel chunks ( splitting the number of records as evenly as possible, but *not* taking into account the holds per bib ), it takes 21.5 minutes. If we use 10 loops, it takes less then 14 minutes. Test Plan: 1) Generate a huge number of holds ( a few thousand at the minimum ) 2) Run the holds queue builder, use the `time` utility to track how much time it took to run 3) Set HoldsQueueParallelLoopsCount to 10 4) Repeat step 2, note the improvement in speed 5) Experiment with other values for HoldsQueueParallelLoopsCount 6) prove t/db_dependent/HoldsQueue.t -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #123671|0 |1 is obsolete| | --- Comment #6 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 123679 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123679&action=edit Bug 28833: Speed up holds queue builder via parallel processing The holds queue builder can take a very long time to run on systems with many holds. For example, a partner with 124,784 unfilled ( not found ) holds, is taking about 64 minutes to run. If we run that same number of holds in 5 parallel chunks ( splitting the number of records as evenly as possible, but *not* taking into account the holds per bib ), it takes 21.5 minutes. If we use 10 loops, it takes less then 14 minutes. Test Plan: 0) Install the Perl library Parallel::Loops 1) Generate a huge number of holds ( a few thousand at the minimum ) 2) Run the holds queue builder, use the `time` utility to track how much time it took to run 3) Set HoldsQueueParallelLoopsCount to 10 4) Repeat step 2, note the improvement in speed 5) Experiment with other values for HoldsQueueParallelLoopsCount 6) prove t/db_dependent/HoldsQueue.t -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #123679|0 |1 is obsolete| | --- Comment #7 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 123688 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123688&action=edit Bug 28833: Speed up holds queue builder via parallel processing The holds queue builder can take a very long time to run on systems with many holds. For example, a partner with 124,784 unfilled ( not found ) holds, is taking about 64 minutes to run. If we run that same number of holds in 5 parallel chunks ( splitting the number of records as evenly as possible, but *not* taking into account the holds per bib ), it takes 21.5 minutes. If we use 10 loops, it takes less then 14 minutes. Test Plan: 0) Install the Perl library Parallel::ForkManager 1) Generate a huge number of holds ( a few thousand at the minimum ) 2) Run the holds queue builder, use the `time` utility to track how much time it took to run 3) Set HoldsQueueParallelLoopsCount to 10 4) Repeat step 2, note the improvement in speed 5) Experiment with other values for HoldsQueueParallelLoopsCount 6) prove t/db_dependent/HoldsQueue.t -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #123688|0 |1 is obsolete| | --- Comment #8 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 123689 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123689&action=edit Bug 28833: Speed up holds queue builder via parallel processing The holds queue builder can take a very long time to run on systems with many holds. For example, a partner with 124,784 unfilled ( not found ) holds, is taking about 64 minutes to run. If we run that same number of holds in 5 parallel chunks ( splitting the number of records as evenly as possible, but *not* taking into account the holds per bib ), it takes 21.5 minutes. If we use 10 loops, it takes less then 14 minutes. Test Plan: 0) Install the Perl library Parallel::ForkManager 1) Generate a huge number of holds ( a few thousand at the minimum ) 2) Run the holds queue builder, use the `time` utility to track how much time it took to run 3) Set HoldsQueueParallelLoopsCount to 10 4) Repeat step 2, note the improvement in speed 5) Experiment with other values for HoldsQueueParallelLoopsCount 6) prove t/db_dependent/HoldsQueue.t -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au --- Comment #9 from David Cook <dcook@prosentient.com.au> --- This is an interesting one! On one hand, I like the use of a system preference to allow the value to be set on a per-instance basis. On the other hand, letting a library define an unrestricted number of forks for a script seems potentially dangerous, especially if you have multiple Koha instances on the same server. I think this change needs more sanity checks. Some options: 1) A CLI override that doesn't let the $loops be beyond X. This allows a sysadmin to constrain the resource usage 2) Perhaps the size of $bibs_with_pending_requests should correlate with the number of forks up to a maximum. (If you only have 10 bibs but your $loops is 10, it doesn't make sense to have 10 forks of 1 bib each. ) 3) Maybe we should limit nproc in /etc/security/limits.d on a per-user basis (but maybe this is best left to local sysadmins and this is a very blunt instrument) Actually, we should also do a int() check on $loops before feeding it to Parallel::ForkManager, since it is user-provided input. (I suppose a non-number would just cause a fatal error on a $loops > 1 check, but better to not fork than to fail.) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 --- Comment #10 from David Cook <dcook@prosentient.com.au> --- Another option might be to use koha-conf.xml instead of a system preference. I think it's important for it to be a configurable value per-instance, but I'm opposed to the idea of library staff being able to determine server resource usage. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 --- Comment #11 from David Cook <dcook@prosentient.com.au> --- The following is totally out of scope but inspired by the above: What if default values in koha-conf.xml were determined based on a "profile" provided for koha-create? By default, we'd have permissive values, so that inexperienced people could have the most power possible through the Web UI, since they might not have a sysadmin to update the backend. But we'd have a more restrictive profile that vendors and more experienced people could use to have more locked down/secure Koha instances. I'm trying to think of an approach that could make everyone happy... -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 --- Comment #12 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 123783 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123783&action=edit Bug 28833: (follow-up) Move loops count control to koha conf -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 --- Comment #13 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 123784 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123784&action=edit Bug 28833: (follow-up) Only spawn multiple processes if there are more records than loops specified -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #123783|0 |1 is obsolete| | Attachment #123784|0 |1 is obsolete| | --- Comment #14 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 123785 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123785&action=edit Bug 28833: (follow-up) Move loops count control to koha conf -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 --- Comment #15 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 123786 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123786&action=edit Bug 28833: (follow-up) Only spawn multiple processes if there are more records than loops specified -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 --- Comment #16 from David Cook <dcook@prosentient.com.au> --- Comment on attachment 123786 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123786 Bug 28833: (follow-up) Only spawn multiple processes if there are more records than loops specified Review of attachment 123786: --> (https://bugs.koha-community.org/bugzilla3/page.cgi?id=splinter.html&bug=28833&attachment=123786) ----------------------------------------------------------------- ::: C4/HoldsQueue.pm @@ +201,4 @@
# Split the list of bibs into groups to run in parallel if ( $loops > 1 ) { my $bibs_per_chunk = ceil( scalar @$bibs_with_pending_requests / $loops ); + $bibs_per_chunk = @$bibs_with_pending_requests if @$bibs_with_pending_requests <= $bibs_per_chunk;
I don't understand this logic. scalar @$bibs_with_pending_requests = 10 $loops = 10 $bibs_per_chunk = 1 $bibs_per_chunk would still be 1. Maybe I'm confused but I can't think of a case where $bibs_per_chunk could be greater than scalar @$bibs_with_pending_requests. -- I was thinking we'd do the $loops would be changed to $max_loops, and $loops would be based on an arbitrary ratio. Perhaps 1000 bibs per 1 loop. So $loops = scalar @$bibs_with_pending_requests / 1000. If $loops < 1, then we don't fork. If $loops > 1, we double-check that we're not exceeding our maximum number of loops. For example: $max_loops = 4 scalar @$bibs_with_pending_requests = 10000 $loops = 10 We throttle $loops down from 10 to 4, because 4 is our maximum. Likewise, if @$bibs_with_pending_requests = 2000, we're only using 2 forks instead of our maximum of 4, so we're not using more resources than necessary. -- Alternatively, we could have an arbitrary threshold of 1000. Below that and we use $loops = 1. Above that, we use the $loops value supplied from config. I'm just trying to avoid scenarios where you have 50 Koha instances each spinning up 10 child processes to process 2 requests each. But maybe I'm just prematurely optimizing. I suppose most instances will have the default of 1 anyway. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 --- Comment #17 from David Cook <dcook@prosentient.com.au> --- Comment on attachment 123689 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123689 Bug 28833: Speed up holds queue builder via parallel processing Review of attachment 123689: --> (https://bugs.koha-community.org/bugzilla3/page.cgi?id=splinter.html&bug=28833&attachment=123689) ----------------------------------------------------------------- ::: C4/HoldsQueue.pm @@ +210,5 @@
+ + DATA_LOOP: + foreach my $chunk (@chunks) { + my $pid = $pm->start and next DATA_LOOP; + _ProcessBiblios($chunk);
I notice that the other _ProcessBiblios has arguments for $branches_to_use and $transport_cost_matrix. I'm guessing those should be included here too? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #123689|0 |1 is obsolete| | Attachment #123785|0 |1 is obsolete| | Attachment #123786|0 |1 is obsolete| | --- Comment #18 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 123996 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123996&action=edit Bug 28833: Speed up holds queue builder via parallel processing The holds queue builder can take a very long time to run on systems with many holds. For example, a partner with 124,784 unfilled ( not found ) holds, is taking about 64 minutes to run. If we run that same number of holds in 5 parallel chunks ( splitting the number of records as evenly as possible, but *not* taking into account the holds per bib ), it takes 21.5 minutes. If we use 10 loops, it takes less then 14 minutes. Test Plan: 0) Install the Perl library Parallel::ForkManager 1) Generate a huge number of holds ( a few thousand at the minimum ) 2) Run the holds queue builder, use the `time` utility to track how much time it took to run 3) Set HoldsQueueParallelLoopsCount to 10 4) Repeat step 2, note the improvement in speed 5) Experiment with other values for HoldsQueueParallelLoopsCount 6) prove t/db_dependent/HoldsQueue.t -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 --- Comment #19 from Kyle M Hall <kyle@bywatersolutions.com> --- (In reply to David Cook from comment #17)
Comment on attachment 123689 [details] [review] Bug 28833: Speed up holds queue builder via parallel processing
Review of attachment 123689 [details] [review]: -----------------------------------------------------------------
::: C4/HoldsQueue.pm @@ +210,5 @@
+ + DATA_LOOP: + foreach my $chunk (@chunks) { + my $pid = $pm->start and next DATA_LOOP; + _ProcessBiblios($chunk);
I notice that the other _ProcessBiblios has arguments for $branches_to_use and $transport_cost_matrix. I'm guessing those should be included here too?
Good catch! I've fixed that issue and squashed the patches down for ease of use. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 --- Comment #20 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 126200 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=126200&action=edit Bug 28833: Balance the number of holds each chunk contains -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #123996|0 |1 is obsolete| | Attachment #126200|0 |1 is obsolete| | --- Comment #21 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 126202 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=126202&action=edit Bug 28833: Speed up holds queue builder via parallel processing The holds queue builder can take a very long time to run on systems with many holds. For example, a partner with 124,784 unfilled ( not found ) holds, is taking about 64 minutes to run. If we run that same number of holds in 5 parallel chunks ( splitting the number of records as evenly as possible, but *not* taking into account the holds per bib ), it takes 21.5 minutes. If we use 10 loops, it takes less then 14 minutes. Test Plan: 0) Install the Perl library Parallel::ForkManager 1) Generate a huge number of holds ( a few thousand at the minimum ) 2) Run the holds queue builder, use the `time` utility to track how much time it took to run 3) Set HoldsQueueParallelLoopsCount to 10 4) Repeat step 2, note the improvement in speed 5) Experiment with other values for HoldsQueueParallelLoopsCount 6) prove t/db_dependent/HoldsQueue.t -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #126202|0 |1 is obsolete| | --- Comment #22 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 126216 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=126216&action=edit Bug 28833: Speed up holds queue builder via parallel processing The holds queue builder can take a very long time to run on systems with many holds. For example, a partner with 124,784 unfilled ( not found ) holds, is taking about 64 minutes to run. If we run that same number of holds in 5 parallel chunks ( splitting the number of records as evenly as possible, but *not* taking into account the holds per bib ), it takes 21.5 minutes. If we use 10 loops, it takes less then 14 minutes. Test Plan: 0) Install the Perl library Parallel::ForkManager 1) Generate a huge number of holds ( a few thousand at the minimum ) 2) Run the holds queue builder, use the `time` utility to track how much time it took to run 3) Set HoldsQueueParallelLoopsCount to 10 4) Repeat step 2, note the improvement in speed 5) Experiment with other values for HoldsQueueParallelLoopsCount 6) prove t/db_dependent/HoldsQueue.t -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Patch doesn't apply CC| |david@davidnind.com --- Comment #23 from David Nind <david@davidnind.com> --- As expected, since this has been in the sign off queue for a while, the patch no longer applies 8-(... Is there a way to generate a large number of holds? Either an SQL query or a script. kohadev-koha@kohadevbox:koha(bz28833)$ git bz apply 28833 Bug 28833 - Speed up holds queue builder via parallel processing 126216 - Bug 28833: Speed up holds queue builder via parallel processing Apply? [(y)es, (n)o, (i)nteractive] y Applying: Bug 28833: Speed up holds queue builder via parallel processing Using index info to reconstruct a base tree... M C4/HoldsQueue.pm M debian/templates/koha-conf-site.xml.in M etc/koha-conf.xml M misc/cronjobs/holds/build_holds_queue.pl Falling back to patching base and 3-way merge... Auto-merging misc/cronjobs/holds/build_holds_queue.pl CONFLICT (content): Merge conflict in misc/cronjobs/holds/build_holds_queue.pl Auto-merging etc/koha-conf.xml CONFLICT (content): Merge conflict in etc/koha-conf.xml Auto-merging debian/templates/koha-conf-site.xml.in CONFLICT (content): Merge conflict in debian/templates/koha-conf-site.xml.in Auto-merging C4/HoldsQueue.pm CONFLICT (content): Merge conflict in C4/HoldsQueue.pm error: Failed to merge in the changes. Patch failed at 0001 Bug 28833: Speed up holds queue builder via parallel processing -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #126216|0 |1 is obsolete| | --- Comment #24 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 167349 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=167349&action=edit Bug 28833: Speed up holds queue builder via parallel processing The holds queue builder can take a very long time to run on systems with many holds. For example, a partner with 124,784 unfilled ( not found ) holds, is taking about 64 minutes to run. If we run that same number of holds in 5 parallel chunks ( splitting the number of records as evenly as possible, but *not* taking into account the holds per bib ), it takes 21.5 minutes. If we use 10 loops, it takes less then 14 minutes. Test Plan: 0) Install the Perl library Parallel::ForkManager 1) Generate a huge number of holds ( a few thousand at the minimum ) 2) Run the holds queue builder, use the `time` utility to track how much time it took to run 3) Set HoldsQueueParallelLoopsCount to 10 4) Repeat step 2, note the improvement in speed 5) Experiment with other values for HoldsQueueParallelLoopsCount 6) prove t/db_dependent/HoldsQueue.t -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 --- Comment #25 from David Nind <david@davidnind.com> --- Thanks Kyle for making the patch apply! Is there an easy way to generate a large number of holds? Either an SQL query or a script. David -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 --- Comment #26 from Kyle M Hall <kyle@bywatersolutions.com> --- (In reply to David Nind from comment #25)
Thanks Kyle for making the patch apply!
Is there an easy way to generate a large number of holds? Either an SQL query or a script.
David
This script will generate 100 holds: #!/usr/bin/perl use t::lib::TestBuilder; my $builder = t::lib::TestBuilder->new; for ( my $i = 0 ; $i < 100 ; $i++ ) { my $hold = $builder->build( { source => 'Reserve' } ); } Hope this helps! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |martin.renvoize@ptfs-europe |y.org |.com CC| |martin.renvoize@ptfs-europe | |.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Brendan Lawlor <blawlor@clamsnet.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |blawlor@clamsnet.org --- Comment #27 from Brendan Lawlor <blawlor@clamsnet.org> --- Test notes:
This script will generate 100 holds:
#!/usr/bin/perl use t::lib::TestBuilder; my $builder = t::lib::TestBuilder->new;
for ( my $i = 0 ; $i < 100 ; $i++ ) { my $hold = $builder->build( { source => 'Reserve' } ); }
I ran this script 10+ times. I noticed that in addition to holds it also makes new patrons, branches, bibs and itemtypes, probably other stuff too. cool! I could see 1000+ records in the reserves table, but if I went to a patron record where a hold was created by the script the holds table wouldn't load, with error 500. I wasn't able to build the holds queue. Also the bib detail page returned the error: DBIC result _type isn't of the _type BiblioMetadata at /kohadevbox/koha/catalogue/detail.pl line 105. at /usr/lib/x86_64-linux-gnu/perl-base/Carp.pm line 289 I think the next step needs to be to reindex for the new bibs, so I tried: koha-rebuild-zebra -f --force -v kohadev Something is not working to reindex the bibs created by the script though, once it gets past the ktd default bibs it returns errors like .error retrieving biblio 3000 at /kohadevbox/koha/misc/migration_tools/rebuild_zebra.pl line 706. Am I missing a step in the reindexing part? What step/s are required after generating the test data? I was able to install Parallel::ForkManager and use the time utility, so I will be able to test again once I can resolve the data issue. Thanks! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Patch doesn't apply --- Comment #28 from David Nind <david@davidnind.com> --- The patch no longer applies for me 8-(... git bz apply 28833 Bug 28833 - Speed up holds queue builder via parallel processing 167349 - Bug 28833: Speed up holds queue builder via parallel processing Apply? [(y)es, (n)o, (i)nteractive] y Applying: Bug 28833: Speed up holds queue builder via parallel processing Using index info to reconstruct a base tree... M C4/HoldsQueue.pm M debian/templates/koha-conf-site.xml.in M etc/koha-conf.xml M misc/cronjobs/holds/build_holds_queue.pl Falling back to patching base and 3-way merge... Auto-merging misc/cronjobs/holds/build_holds_queue.pl CONFLICT (content): Merge conflict in misc/cronjobs/holds/build_holds_queue.pl Auto-merging etc/koha-conf.xml Auto-merging debian/templates/koha-conf-site.xml.in Auto-merging C4/HoldsQueue.pm CONFLICT (content): Merge conflict in C4/HoldsQueue.pm error: Failed to merge in the changes. Patch failed at 0001 Bug 28833: Speed up holds queue builder via parallel processing -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC|martin.renvoize@ptfs-europe | |.com | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 --- Comment #29 from Martin Renvoize (ashimema) <martin.renvoize@ptfs-europe.com> --- Is this one still relevant Kyle? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #167349|0 |1 is obsolete| | --- Comment #30 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 173045 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=173045&action=edit Bug 28833: Speed up holds queue builder via parallel processing The holds queue builder can take a very long time to run on systems with many holds. For example, a partner with 124,784 unfilled ( not found ) holds, is taking about 64 minutes to run. If we run that same number of holds in 5 parallel chunks ( splitting the number of records as evenly as possible, but *not* taking into account the holds per bib ), it takes 21.5 minutes. If we use 10 loops, it takes less then 14 minutes. Test Plan: 0) Install the Perl library Parallel::ForkManager 1) Generate a huge number of holds ( a few thousand at the minimum ) 2) Run the holds queue builder, use the `time` utility to track how much time it took to run 3) Set HoldsQueueParallelLoopsCount to 10 4) Repeat step 2, note the improvement in speed 5) Experiment with other values for HoldsQueueParallelLoopsCount 6) prove t/db_dependent/HoldsQueue.t -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #173045|0 |1 is obsolete| | --- Comment #31 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 173046 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=173046&action=edit Bug 28833: Speed up holds queue builder via parallel processing The holds queue builder can take a very long time to run on systems with many holds. For example, a partner with 124,784 unfilled ( not found ) holds, is taking about 64 minutes to run. If we run that same number of holds in 5 parallel chunks ( splitting the number of records as evenly as possible, but *not* taking into account the holds per bib ), it takes 21.5 minutes. If we use 10 loops, it takes less then 14 minutes. Test Plan: 0) Install the Perl library Parallel::ForkManager 1) Generate a huge number of holds ( a few thousand at the minimum ) 2) Run the holds queue builder, use the `time` utility to track how much time it took to run 3) Set HoldsQueueParallelLoopsCount to 10 4) Repeat step 2, note the improvement in speed 5) Experiment with other values for HoldsQueueParallelLoopsCount 6) prove t/db_dependent/HoldsQueue.t -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Martin Renvoize (ashimema) <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #173046|0 |1 is obsolete| | --- Comment #32 from Martin Renvoize (ashimema) <martin.renvoize@ptfs-europe.com> --- Created attachment 173262 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=173262&action=edit Bug 28833: Speed up holds queue builder via parallel processing The holds queue builder can take a very long time to run on systems with many holds. For example, a partner with 124,784 unfilled ( not found ) holds, is taking about 64 minutes to run. If we run that same number of holds in 5 parallel chunks ( splitting the number of records as evenly as possible, but *not* taking into account the holds per bib ), it takes 21.5 minutes. If we use 10 loops, it takes less then 14 minutes. Test Plan: 0) Install the Perl library Parallel::ForkManager 1) Generate a huge number of holds ( a few thousand at the minimum ) 2) Run the holds queue builder, use the `time` utility to track how much time it took to run 3) Set HoldsQueueParallelLoopsCount to 10 4) Repeat step 2, note the improvement in speed 5) Experiment with other values for HoldsQueueParallelLoopsCount 6) prove t/db_dependent/HoldsQueue.t Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Martin Renvoize (ashimema) <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Martin Renvoize (ashimema) <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA --- Comment #33 from Martin Renvoize (ashimema) <martin.renvoize@ptfs-europe.com> --- This works well and I see no regressions. Passing QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |release-notes-needed Version|unspecified |Main --- Comment #34 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Wow, old style database update, reformatted to use the newer template on review. I verified that the holds queue is still built running the script, but didn't test the parallel processing. Trusting the process :) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |24.11.00 released in| | Status|Passed QA |Pushed to main -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 --- Comment #35 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Pushed for 24.11! Well done everyone, thank you! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Philip Orr <philip.orr@lmscloud.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the| |This enhancement adds the release notes| |ability to increase the | |amount of parallel | |calculations used to create | |the Holds Queue. The amount | |of calculations or "loops" | |done in parallel is set in | |the new system preference | |'HoldsQueueParallelLoopsCou | |nt'. This enhancement will | |allow faster holds queue | |building for libraries with | |a very large amount of | |holds. Note that increasing | |the value in the system | |preference also causes Koha | |to use more resources when | |building the holds queue. Keywords|release-notes-needed | CC| |philip.orr@lmscloud.de --- Comment #36 from Philip Orr <philip.orr@lmscloud.de> --- added release notes -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Lucas Gass (lukeg) <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED CC| |lucas@bywatersolutions.com Status|Pushed to main |RESOLVED --- Comment #37 from Lucas Gass (lukeg) <lucas@bywatersolutions.com> --- Enhancement, no 24.05.x backport -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |Needs documenting Resolution|FIXED |--- --- Comment #38 from David Nind <david@davidnind.com> --- Changed status to needs documenting - new system preference that needs documenting. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Aude Charillon <aude.charillon@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Documentation| |Aude Charillon contact| | Documentation| |https://gitlab.com/koha-com submission| |munity/koha-manual/-/merge_ | |requests/1093 CC| |aude.charillon@openfifth.co | |.uk -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Aude Charillon <aude.charillon@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED Status|Needs documenting |RESOLVED --- Comment #39 from Aude Charillon <aude.charillon@openfifth.co.uk> --- New system preference added to the Koha Manual. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |41781 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41781 [Bug 41781] Holds queue builder ( build_holds_queue.pl ) fails if HoldsQueueParallelLoopsCount is greater than 1 -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org