[Bug 28413] New: background job worker is running with all the modules in RAM
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 Bug ID: 28413 Summary: background job worker is running with all the modules in RAM Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: ASSIGNED Severity: enhancement Priority: P5 - low Component: Architecture, internals, and plumbing Assignee: jonathan.druart+koha@gmail.com Reporter: jonathan.druart+koha@gmail.com QA Contact: testopia@bugs.koha-community.org It's loading the modules in RAM then forks. Then do not need it to have them loaded in the parent process (the worker/daemon). -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=27434 Depends on| |27756, 22417 Blocks| |28410 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 [Bug 22417] Add a task queue https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27756 [Bug 27756] background_jobs_worker.pl is memory inefficient https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28410 [Bug 28410] Reduce memory footprint -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 --- Comment #1 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 121274 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=121274&action=edit Bug 28413: Reduce background job worker memory footprint It's loading the modules in RAM then forks. Then do not need it to have them loaded in the parent process (the worker/daemon). -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |In Discussion -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au --- Comment #2 from David Cook <dcook@prosentient.com.au> --- I think that this is an improvement, but it would be nice to not have the job type hard-coded into the process method. Personally, I'd like the job type to either be the fully qualified module name (e.g. Koha::BackgroundJob::BatchUpdateBiblio) or a shortened version (e.g. BatchUpdateBiblio), so that Koha::BackgroundJob::process() could to "load $job_type;" (using Module::Load) and "$job_type->process($args)". But perhaps that's out of scope for this change.. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 --- Comment #3 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Yes, that's for bug 27434. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Discussion |Needs Signoff --- Comment #4 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Test plan: Confirm the difference of memory allocated to the background job worker with and without the patch. Confirm the it's still working correctly after you applied the patch (batch modify some biblios for instance). -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |rel_21_11_candidate, | |RM_priority -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |tomascohen@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 --- Comment #5 from David Cook <dcook@prosentient.com.au> --- On my radar but I'm catching up on a backlog atm. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #121274|0 |1 is obsolete| | --- Comment #6 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 125392 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=125392&action=edit Bug 28413: Reduce background job worker memory footprint It's loading the modules in RAM then forks. Then do not need it to have them loaded in the parent process (the worker/daemon). Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off CC| |martin.renvoize@ptfs-europe | |.com --- Comment #7 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Nice little improvement.. I too am looking forward to the next steps but I see no reason to hold this step up. Signing off.. tempted to go straight PQA as it's a trivially simple piece of code to understand.. but I'll er on the side of caution and poke a fellow QA team person to push it over the edge. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Patch doesn't apply --- Comment #8 from David Cook <dcook@prosentient.com.au> --- Sorry for taking so long on this one. Patch doesn't apply anymore. As a side note, before applying the patch, it looks like VIRT 241.2m RES 175.3m SHR 19.2m, according to top. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #125392|0 |1 is obsolete| | --- Comment #9 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 125941 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=125941&action=edit Bug 28413: Reduce background job worker memory footprint It's loading the modules in RAM then forks. Then do not need it to have them loaded in the parent process (the worker/daemon). Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 --- Comment #10 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- VIRT RES SH 200880 145832 25496 # Before 104196 60224 15192 # After -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 --- Comment #11 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- VIRT RES SHR 200880 145832 25496 # Before 104196 60224 15192 # After -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|RM_priority | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david@davidnind.com Status|Needs Signoff |Patch doesn't apply --- Comment #12 from David Nind <david@davidnind.com> --- Patch no longer applies 8-(... Applying: Bug 28413: Reduce background job worker memory footprint Using index info to reconstruct a base tree... M Koha/BackgroundJob.pm Falling back to patching base and 3-way merge... Auto-merging Koha/BackgroundJob.pm CONFLICT (content): Merge conflict in Koha/BackgroundJob.pm error: Failed to merge in the changes. Patch failed at 0001 Bug 28413: Reduce background job worker memory footprint -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #125941|0 |1 is obsolete| | --- Comment #13 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 127081 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=127081&action=edit Bug 28413: Reduce background job worker memory footprint It's loading the modules in RAM then forks. Then do not need it to have them loaded in the parent process (the worker/daemon). Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off --- Comment #14 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- This has been signed off actually and is waiting for QA. Thanks David, you can still test it if you want to! :) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 --- Comment #15 from David Nind <david@davidnind.com> --- I wondered about that (whether it was already signed off)... I think Martin signing it off is more than sufficient! 8-) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #127081|0 |1 is obsolete| | --- Comment #16 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Created attachment 127329 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=127329&action=edit Bug 28413: Reduce background job worker memory footprint It's loading the modules in RAM then forks. Then do not need it to have them loaded in the parent process (the worker/daemon). Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |victor@tuxayo.net --- Comment #17 from Victor Grousset/tuxayo <victor@tuxayo.net> --- works (tested on bib record, auth, item and hold cancel, ram usage decreased), QA script happy, code look ok. Passed! It's surprising that these classes take dozens of MB of RAM to load. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 --- Comment #18 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- (In reply to Victor Grousset/tuxayo from comment #17)
works (tested on bib record, auth, item and hold cancel, ram usage decreased), QA script happy, code look ok.
Passed!
Thanks!
It's surprising that these classes take dozens of MB of RAM to load.
If you load one, you load all. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to master Version(s)| |21.11.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 --- Comment #19 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Pushed to master for 21.11, thanks to everybody involved! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|rel_21_11_candidate | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED CC| |kyle@bywatersolutions.com Status|Pushed to master |RESOLVED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 Andrew Nugged <nugged@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nugged@gmail.com --- Comment #20 from Andrew Nugged <nugged@gmail.com> --- in https://git.koha-community.org/Koha-community/Koha/src/branch/master/Koha/Ba... my $item = Koha::Items->find($record_id) || next; it dies because was lost "use Koha::Items;" like here: https://git.koha-community.org/Koha-community/Koha/src/branch/master/Koha/Ba... since now packages are not included all at once, like before this ticket, when all modules were loaded by neighbors so the error was hidden. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 --- Comment #21 from Andrew Nugged <nugged@gmail.com> --- Created attachment 127598 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=127598&action=edit Bug 28413: (QA follow-up) Fix for missing use Koha::Items -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |additional_work_needed -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|additional_work_needed | --- Comment #22 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- (In reply to Andrew Nugged from comment #21)
Created attachment 127598 [details] [review] Bug 28413: (QA follow-up) Fix for missing use Koha::Items
Pushed to master, thanks Andrew! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 Bug 28413 depends on bug 22417, which changed state. Bug 22417 Summary: Add a task queue https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to master |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 Bug 28413 depends on bug 27756, which changed state. Bug 27756 Summary: background_jobs_worker.pl is memory inefficient https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27756 What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to master |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on|22417 | Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 [Bug 22417] Add a task queue -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |32561 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32561 [Bug 32561] background job worker is still running with all the modules in RAM -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 --- Comment #23 from David Cook <dcook@prosentient.com.au> --- I reckon we could keep reducing the memory footprint on background_jobs_worker.pl. When you have a lot of instances with at least 2 of those workers, the memory usage does add up. -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org