[Bug 36549] New: es_indexer_daemon.pl leaks memory
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 Bug ID: 36549 Summary: es_indexer_daemon.pl leaks memory Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Architecture, internals, and plumbing Assignee: koha-bugs@lists.koha-community.org Reporter: dcook@prosentient.com.au QA Contact: testopia@bugs.koha-community.org background_jobs_worker.pl runs its tasks in child processes which return their memory to the OS when they complete. es_indexer_daemon.pl runs everything in itself which means if it needs to allocate a large amount of memory for an indexing job, it will hold on to that memory allocation for its entire process life. -- I've got an es_indexer_daemon.pl which is using over 1GB of memory. Once it's done with its indexing activity, I'm going to restart it to get it back down to a more reasonable amount of allocated memory. I reckon we should be forking off child processes to do the worker. We don't need many. We just need 1 that is short lived so that it can return the memory back to the OS. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |LATER Status|NEW |RESOLVED --- Comment #1 from David Cook <dcook@prosentient.com.au> --- In prod, we seem to restart the services often enough this isn't a problem, so going to close this one for now. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|LATER |--- Status|RESOLVED |REOPENED --- Comment #2 from David Cook <dcook@prosentient.com.au> --- Re-opening this one as it looks like it's still an issue, and Elasticsearch usage is becoming more prevalent... -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 --- Comment #3 from David Cook <dcook@prosentient.com.au> --- Note also that since es_indexer_daemon.pl doesn't fork workers, it also seems to have a L1 cache issue... -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=39769 -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|--- |Small patch Assignee|koha-bugs@lists.koha-commun |nick@bywatersolutions.com |ity.org | Status|REOPENED |Needs Signoff -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 --- Comment #4 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 203890 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=203890&action=edit Bug 36549: Use Parallel::ForkManager in es_indexer_daemon This patch copies the implementation from the general background worked into the es_indexer daemon Note: There is a 'sleep(10)' in the general indexer and I copied it here, but I don't know why we do it? To test: 1 - Apply the patch 2 - sudo koha-es-indxer --restart kohadev 3 - perl misc/maintenance/touch_all_biblios.pl 4 - Wait a bit 5 - Check the background jobs in the admin interface 6 - Confirm the jobs finish and records are searchable 7 - Test with and without rabbitmq enabled 8 - Sign off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david@davidnind.com --- Comment #5 from David Nind <david@davidnind.com> --- Not sure if I'm getting the expected results for step 7 of the test plan. With rabbitmq stopped, I get: - errors when running perl misc/maintenance/touch_all_biblios.pl - all 367 jobs show as: . Status: New . Progress: null/1 Testing notes (using KTD): 1. Step 2 - command should be: sudo koha-es-indexer --restart kohadev 2. Step 7: I get an error if I stop rabbitmq (sudo -s service rabbitmq-server stop): - More > About Koha > Server information shows: Message broker: Using SQL polling (Fallback, Error connecting to RabbitMQ) - More > About Koha > System informations shows: Impossible to connect to the message broker There is an error when trying to connect to the message broker (RabbitMQ), check the Koha log files. Maybe it is not installed and configured correctly? Contact your system administrator. - Error message repeated for every record I think when running: perl misc/maintenance/touch_all_biblios.pl Cannot connect to broker (Failed to connect: Error connecting to localhost:61613: Connection refused at /usr/share/perl5/Net/Stomp.pm line 27.; giving up at /usr/share/perl5/Net/Stomp.pm line 27. ) at /kohadevbox/koha/Koha/BackgroundJob.pm line 100. - All 367 jobs show as: . Status: New . Progress: null/1 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 --- Comment #6 from David Cook <dcook@prosentient.com.au> --- Comment on attachment 203890 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=203890 Bug 36549: Use Parallel::ForkManager in es_indexer_daemon Review of attachment 203890: --> (https://bugs.koha-community.org/bugzilla3/page.cgi?id=splinter.html&bug=36549&attachment=203890) ----------------------------------------------------------------- ::: misc/workers/es_indexer_daemon.pl @@ +216,5 @@
+ $pm->start and next; + srand(); # ensure each child process begins with a new seed + commit( @{$commit_jobs} ); + $pm->finish; + sleep 10;
So why are you adding a sleep here? In /kohadevbox/koha/misc/workers/background_jobs_worker.pl, the sleep is only used for the database polling. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 --- Comment #7 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- (In reply to David Nind from comment #5)
Not sure if I'm getting the expected results for step 7 of the test plan.
With rabbitmq stopped, I get: - errors when running perl misc/maintenance/touch_all_biblios.pl - all 367 jobs show as: . Status: New . Progress: null/1
Testing notes (using KTD): 1. Step 2 - command should be: sudo koha-es-indexer --restart kohadev 2. Step 7: I get an error if I stop rabbitmq (sudo -s service rabbitmq-server stop): - More > About Koha > Server information shows: Message broker: Using SQL polling (Fallback, Error connecting to RabbitMQ) - More > About Koha > System informations shows: Impossible to connect to the message broker There is an error when trying to connect to the message broker (RabbitMQ), check the Koha log files. Maybe it is not installed and configured correctly? Contact your system administrator. - Error message repeated for every record I think when running: perl misc/maintenance/touch_all_biblios.pl
Cannot connect to broker (Failed to connect: Error connecting to localhost:61613: Connection refused at /usr/share/perl5/Net/Stomp.pm line 27.; giving up at /usr/share/perl5/Net/Stomp.pm line 27. ) at /kohadevbox/koha/Koha/BackgroundJob.pm line 100.
- All 367 jobs show as: . Status: New . Progress: null/1
Did you restart the indexer after stopping rabbit? The errors trying to connect are expected, but new jobs should be done via polling -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 --- Comment #8 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- (In reply to David Cook from comment #6)
Comment on attachment 203890 [details] [review] So why are you adding a sleep here? In /kohadevbox/koha/misc/workers/background_jobs_worker.pl, the sleep is only used for the database polling.
I tried asking around to see why that sleep was added to the script, but didn't get an answer, so I added it in both sections for safety, I can easily remove if we understand the purpose ;-) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #203890|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 --- Comment #9 from David Nind <david@davidnind.com> --- Created attachment 204069 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=204069&action=edit Bug 36549: Use Parallel::ForkManager in es_indexer_daemon This patch copies the implementation from the general background worked into the es_indexer daemon Note: There is a 'sleep(10)' in the general indexer and I copied it here, but I don't know why we do it? To test: 1 - Apply the patch 2 - sudo koha-es-indexer --restart kohadev 3 - perl misc/maintenance/touch_all_biblios.pl 4 - Wait a bit 5 - Check the background jobs in the admin interface 6 - Confirm the jobs finish and records are searchable 7 - Test with and without rabbitmq enabled: . Stop rabbitmq service: sudo -s service rabbitmq-server stop . Repeat steps 2 to 5 8 - Sign off Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 --- Comment #10 from David Nind <david@davidnind.com> --- (In reply to Nick Clemens (kidclamp) from comment #7)
Did you restart the indexer after stopping rabbit? The errors trying to connect are expected, but new jobs should be done via polling
I have now 8-) - now signed off. (I updated the test plan in the commit message to reflect this.) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lisette@bywatersolutions.co | |m QA Contact|testopia@bugs.koha-communit |aleisha@catalyst.net.nz |y.org | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #204069|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 --- Comment #11 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 204191 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=204191&action=edit Bug 36549: Use Parallel::ForkManager in es_indexer_daemon This patch copies the implementation from the general background worked into the es_indexer daemon Note: There is a 'sleep(10)' in the general indexer and I copied it here, but I don't know why we do it? To test: 1 - Apply the patch 2 - sudo koha-es-indexer --restart kohadev 3 - perl misc/maintenance/touch_all_biblios.pl 4 - Wait a bit 5 - Check the background jobs in the admin interface 6 - Confirm the jobs finish and records are searchable 7 - Test with and without rabbitmq enabled: . Stop rabbitmq service: sudo -s service rabbitmq-server stop . Repeat steps 2 to 5 8 - Sign off Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Aleisha Amohia <aleisha@catalyst.net.nz> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|normal |enhancement -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |43016 CC| |andrew@bywatersolutions.com Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43016 [Bug 43016] [OMNIBUS] Server resource protection -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |rel_26_11_candidate Severity|enhancement |normal -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 --- Comment #12 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 205333 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205333&action=edit Bug 36549: JUST FOR TEST - prove sleep(10) worked BEFORE this patch Test plan (apply just this patch): 1) Set the 'SearchEngine' system preference to 'Elasticsearch'. 2) sudo -s service rabbitmq-server stop 3) sudo koha-es-indexer --stop kohadev perl /kohadevbox/koha/misc/workers/es_indexer_daemon.pl 4) Watch the output. Every cycle should print, in order: 'POLL: about to commit ...' 'POLL: about to sleep(10) ...' 'POLL: woke up from sleep(10) ...' 'PARENT: back at top of loop ...' 5) Check the timestamps on 'about to sleep(10)' vs 'woke up from sleep(10)': they should be ~10 seconds apart, every single cycle. This confirms the throttle worked correctly before the patchset here. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 --- Comment #13 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 205360 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205360&action=edit Bug 36549: JUST FOR TEST - prove sleep(10) never runs after $pm->finish Do not push. Apply this on top of the bug 36549 patchset (it builds on that patch, it will not apply standalone). Adds warn() calls around the fork/commit/finish/sleep sequence in both branches to show live that $pm->finish exits the child process, so nothing after it (including sleep 10) ever runs. Test plan (apply just this patch): 1) Set the 'SearchEngine' system preference to 'Elasticsearch'. 2) sudo -s service rabbitmq-server stop 3) sudo koha-es-indexer --stop kohadev perl /kohadevbox/koha/misc/workers/es_indexer_daemon.pl 4) Watch the output, repeated per batch: 'POLL: about to fork ...' 'POLL CHILD: about to commit ...' 'POLL CHILD: about to call finish ...' 'PARENT: back at top of loop ...' 5) Notice 'survived finish()' and 'survived sleep(10)' never print, and a new batch starts every second or so instead of every 10 seconds. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |pedro.amorim@openfifth.co.u | |k Status|Passed QA |Failed QA --- Comment #14 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Hi guys, I've attached 2 patches to be used for testing before and after the original patch here. At best it proves the new sleep(10) is unreachable, at worst it proves that the pre-existing original 10 seconds sleep is no longer here. Can we revisit this? Or confirm if the above is intentional? Thanks in advance. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|aleisha@catalyst.net.nz |testopia@bugs.koha-communit | |y.org -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 --- Comment #15 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 206243 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=206243&action=edit Bug 36549: (QA follow-up) Restore sleep Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 --- Comment #16 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- (In reply to Pedro Amorim (ammopt) from comment #14)
Hi guys, I've attached 2 patches to be used for testing before and after the original patch here. At best it proves the new sleep(10) is unreachable, at worst it proves that the pre-existing original 10 seconds sleep is no longer here.
Can we revisit this? Or confirm if the above is intentional? Thanks in advance.
I restored the sleep as it did get picked up before, I do not have an answer from anyone as to why it is there. I think the safest is to leave it here (and add to rabbit) and remove it on another bug -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |43566 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43566 [Bug 43566] Remove sleep during background es indexing -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |zoho.roboto@bywatersolution | |s.com -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org