[Koha-bugs] [Bug 28833] Speed up holds queue builder via parallel processing

[bugzilla-daemon at bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28833

--- Comment #16 from David Cook <dcook at prosentient.com.au> ---
Comment on attachment 123786
  --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=123786
Bug 28833: (follow-up) Only spawn multiple processes if there are more records
than loops specified

Review of attachment 123786:
 --> (https://bugs.koha-community.org/bugzilla3/page.cgi?id=splinter.html&bug=28833&attachment=123786)
-----------------------------------------------------------------

::: C4/HoldsQueue.pm
@@ +201,4 @@
>      # Split the list of bibs into groups to run in parallel
>      if ( $loops > 1 ) {
>          my $bibs_per_chunk = ceil( scalar @$bibs_with_pending_requests / $loops );
> +        $bibs_per_chunk = @$bibs_with_pending_requests if @$bibs_with_pending_requests <= $bibs_per_chunk;

I don't understand this logic. 

scalar @$bibs_with_pending_requests = 10
$loops = 10
$bibs_per_chunk =  1

$bibs_per_chunk would still be 1.

Maybe I'm confused but I can't think of a case where $bibs_per_chunk could be
greater than scalar @$bibs_with_pending_requests.

--

I was thinking we'd do the $loops would be changed to $max_loops, and $loops
would be based on an arbitrary ratio. Perhaps 1000 bibs per 1 loop. 

So $loops = scalar @$bibs_with_pending_requests / 1000. 

If $loops < 1, then we don't fork. If $loops > 1, we double-check that we're
not exceeding our maximum number of loops. 

For example:
$max_loops = 4 
scalar @$bibs_with_pending_requests = 10000
$loops = 10

We throttle $loops down from 10 to 4, because 4 is our maximum. 

Likewise, if @$bibs_with_pending_requests = 2000, we're only using 2 forks
instead of our maximum of 4, so we're not using more resources than necessary. 

--

Alternatively, we could have an arbitrary threshold of 1000. Below that and we
use $loops = 1. Above that, we use the $loops value supplied from config. 

I'm just trying to avoid scenarios where you have 50 Koha instances each
spinning up 10 child processes to process 2 requests each. 

But maybe I'm just prematurely optimizing. I suppose most instances will have
the default of 1 anyway.

-- 
You are receiving this mail because:
You are watching all bug changes.