[Bug 27756] New: background_jobs_worker.pl is memory inefficient
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27756 Bug ID: 27756 Summary: background_jobs_worker.pl is memory inefficient Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Architecture, internals, and plumbing Assignee: koha-bugs@lists.koha-community.org Reporter: dcook@prosentient.com.au QA Contact: testopia@bugs.koha-community.org Once Perl allocates memory, it doesn't return that memory to the Operating System for the life of that Perl process. If memory is freed within a Perl program (nullifying a variable, for instance), that memory will be re-used within the Perl program, but it will not return the memory. This can cause unexpected excessive memory consumption for long-running Perl programs. misc/background_jobs_worker.pl runs all its code within the same process, so it will never return allocated memory. This means that 1 large background import could cause background_jobs_worker.pl to permanently use a large amount of memory (until it is restarted). The solution is to do the background work within a forked process. I was reminded of this when I read "The worker process will fork a new process for every job that is being processed. This allows for resources such as memory to be returned to the operating system once a job is finished. Perl fork is very fast, so don't worry about the overhead" at https://docs.mojolicious.org/Minion. It's the same premise as bug 27267. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27756 --- Comment #1 from David Cook <dcook@prosentient.com.au> --- Some food for thought at https://stackoverflow.com/questions/9733146/tips-for-keeping-perl-memory-usa... -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27756 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |22417 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 [Bug 22417] Add a task queue -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27756 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=27782 -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27756 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |martin.renvoize@ptfs-europe | |.com --- Comment #2 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Haha, I was thinking along the same lines as you! As an aside, I think forking would also resolve bug 27782 right? -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27756 --- Comment #3 from David Cook <dcook@prosentient.com.au> --- (In reply to Martin Renvoize from comment #2)
Haha, I was thinking along the same lines as you!
There's a reason we're friends. Haha.
As an aside, I think forking would also resolve bug 27782 right?
Yeah, it would resolve bug 27782 too. Double win. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27756 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27756 --- Comment #4 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 117500 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=117500&action=edit Bug 27756: Fork background jobs to prevent memory leak -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27756 Owen Leonard <oleonard@myacpl.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |jonathan.druart@bugs.koha-c |ity.org |ommunity.org -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27756 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #117500|0 |1 is obsolete| | --- Comment #5 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 119625 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=119625&action=edit Bug 27756: Fork background jobs to prevent memory leak Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27756 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off --- Comment #6 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Works as expected to me.. but I'm not an expert in forking so look to someone more familiar with that for a QA. Signing off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27756 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #119625|0 |1 is obsolete| | --- Comment #7 from Julian Maurice <julian.maurice@biblibre.com> --- Created attachment 120661 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=120661&action=edit Bug 27756: Fork background jobs to prevent memory leak Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Julian Maurice <julian.maurice@biblibre.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27756 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA CC| |julian.maurice@biblibre.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27756 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to master Version(s)| |21.05.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27756 --- Comment #8 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Pushed to master for 21.05, thanks to everybody involved! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27756 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m --- Comment #9 from Fridolin Somers <fridolin.somers@biblibre.com> --- Do you think this should be pushed to 20.11.x ? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27756 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |28413 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28413 [Bug 28413] background job worker is running with all the modules in RAM -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27756 --- Comment #10 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- *** Bug 27782 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27756 Bug 27756 depends on bug 22417, which changed state. Bug 22417 Summary: Add a task queue https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to master |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27756 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to master |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org