[Bug 35439] New: Large background jobs can create delays
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35439 Bug ID: 35439 Summary: Large background jobs can create delays Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Architecture, internals, and plumbing Assignee: koha-bugs@lists.koha-community.org Reporter: nick@bywatersolutions.com QA Contact: testopia@bugs.koha-community.org When a large import or other background job is enqueued it can slow down the processing of other jobs. A novel idea would be to have this job break itself into smaller chunks, and enqueue those jobs - this way the queue could continue to advance and the smaller jobs could be interleaved with incoming jobs. See bug 35438 as well -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35439 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |wizzyrea@gmail.com See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=35438 -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35439 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au, | |jonathan.druart+koha@gmail. | |com, | |martin.renvoize@ptfs-europe | |.com, tomascohen@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35439 --- Comment #1 from David Cook <dcook@prosentient.com.au> --- I like the idea of the large task being broken into smaller jobs. However, at the moment, I don't know how that would work in terms of updating the result store and showing the user their progress? -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35439 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |matt.blenkinsop@ptfs-europe | |.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35439 --- Comment #2 from Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com> --- I've actually got a start point for this in a piece of work I'm doing for importing KBART files into ERM. The files can contain 10s of 1000s of titles and quite often breach the max_allowed_packet in the database when you try and store the args in the background_jobs table. I've got some subroutines that chunk those files to make sure that we are never more than 75% of the max_allowed_packet and enqueue chunked jobs based on those acceptable sizes. They are still large and sometimes slow jobs though and they all just get passed to the worker queue in order - how would you envisage the interleaving of jobs? -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
participants (1)
-
bugzilla-daemon@bugs.koha-community.org