[Bug 36549] New: es_indexer_daemon.pl leaks memory
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 Bug ID: 36549 Summary: es_indexer_daemon.pl leaks memory Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Architecture, internals, and plumbing Assignee: koha-bugs@lists.koha-community.org Reporter: dcook@prosentient.com.au QA Contact: testopia@bugs.koha-community.org background_jobs_worker.pl runs its tasks in child processes which return their memory to the OS when they complete. es_indexer_daemon.pl runs everything in itself which means if it needs to allocate a large amount of memory for an indexing job, it will hold on to that memory allocation for its entire process life. -- I've got an es_indexer_daemon.pl which is using over 1GB of memory. Once it's done with its indexing activity, I'm going to restart it to get it back down to a more reasonable amount of allocated memory. I reckon we should be forking off child processes to do the worker. We don't need many. We just need 1 that is short lived so that it can return the memory back to the OS. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |LATER Status|NEW |RESOLVED --- Comment #1 from David Cook <dcook@prosentient.com.au> --- In prod, we seem to restart the services often enough this isn't a problem, so going to close this one for now. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|LATER |--- Status|RESOLVED |REOPENED --- Comment #2 from David Cook <dcook@prosentient.com.au> --- Re-opening this one as it looks like it's still an issue, and Elasticsearch usage is becoming more prevalent... -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 --- Comment #3 from David Cook <dcook@prosentient.com.au> --- Note also that since es_indexer_daemon.pl doesn't fork workers, it also seems to have a L1 cache issue... -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=39769 -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|--- |Small patch Assignee|koha-bugs@lists.koha-commun |nick@bywatersolutions.com |ity.org | Status|REOPENED |Needs Signoff -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 --- Comment #4 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 203890 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=203890&action=edit Bug 36549: Use Parallel::ForkManager in es_indexer_daemon This patch copies the implementation from the general background worked into the es_indexer daemon Note: There is a 'sleep(10)' in the general indexer and I copied it here, but I don't know why we do it? To test: 1 - Apply the patch 2 - sudo koha-es-indxer --restart kohadev 3 - perl misc/maintenance/touch_all_biblios.pl 4 - Wait a bit 5 - Check the background jobs in the admin interface 6 - Confirm the jobs finish and records are searchable 7 - Test with and without rabbitmq enabled 8 - Sign off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david@davidnind.com --- Comment #5 from David Nind <david@davidnind.com> --- Not sure if I'm getting the expected results for step 7 of the test plan. With rabbitmq stopped, I get: - errors when running perl misc/maintenance/touch_all_biblios.pl - all 367 jobs show as: . Status: New . Progress: null/1 Testing notes (using KTD): 1. Step 2 - command should be: sudo koha-es-indexer --restart kohadev 2. Step 7: I get an error if I stop rabbitmq (sudo -s service rabbitmq-server stop): - More > About Koha > Server information shows: Message broker: Using SQL polling (Fallback, Error connecting to RabbitMQ) - More > About Koha > System informations shows: Impossible to connect to the message broker There is an error when trying to connect to the message broker (RabbitMQ), check the Koha log files. Maybe it is not installed and configured correctly? Contact your system administrator. - Error message repeated for every record I think when running: perl misc/maintenance/touch_all_biblios.pl Cannot connect to broker (Failed to connect: Error connecting to localhost:61613: Connection refused at /usr/share/perl5/Net/Stomp.pm line 27.; giving up at /usr/share/perl5/Net/Stomp.pm line 27. ) at /kohadevbox/koha/Koha/BackgroundJob.pm line 100. - All 367 jobs show as: . Status: New . Progress: null/1 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 --- Comment #6 from David Cook <dcook@prosentient.com.au> --- Comment on attachment 203890 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=203890 Bug 36549: Use Parallel::ForkManager in es_indexer_daemon Review of attachment 203890: --> (https://bugs.koha-community.org/bugzilla3/page.cgi?id=splinter.html&bug=36549&attachment=203890) ----------------------------------------------------------------- ::: misc/workers/es_indexer_daemon.pl @@ +216,5 @@
+ $pm->start and next; + srand(); # ensure each child process begins with a new seed + commit( @{$commit_jobs} ); + $pm->finish; + sleep 10;
So why are you adding a sleep here? In /kohadevbox/koha/misc/workers/background_jobs_worker.pl, the sleep is only used for the database polling. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 --- Comment #7 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- (In reply to David Nind from comment #5)
Not sure if I'm getting the expected results for step 7 of the test plan.
With rabbitmq stopped, I get: - errors when running perl misc/maintenance/touch_all_biblios.pl - all 367 jobs show as: . Status: New . Progress: null/1
Testing notes (using KTD): 1. Step 2 - command should be: sudo koha-es-indexer --restart kohadev 2. Step 7: I get an error if I stop rabbitmq (sudo -s service rabbitmq-server stop): - More > About Koha > Server information shows: Message broker: Using SQL polling (Fallback, Error connecting to RabbitMQ) - More > About Koha > System informations shows: Impossible to connect to the message broker There is an error when trying to connect to the message broker (RabbitMQ), check the Koha log files. Maybe it is not installed and configured correctly? Contact your system administrator. - Error message repeated for every record I think when running: perl misc/maintenance/touch_all_biblios.pl
Cannot connect to broker (Failed to connect: Error connecting to localhost:61613: Connection refused at /usr/share/perl5/Net/Stomp.pm line 27.; giving up at /usr/share/perl5/Net/Stomp.pm line 27. ) at /kohadevbox/koha/Koha/BackgroundJob.pm line 100.
- All 367 jobs show as: . Status: New . Progress: null/1
Did you restart the indexer after stopping rabbit? The errors trying to connect are expected, but new jobs should be done via polling -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 --- Comment #8 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- (In reply to David Cook from comment #6)
Comment on attachment 203890 [details] [review] So why are you adding a sleep here? In /kohadevbox/koha/misc/workers/background_jobs_worker.pl, the sleep is only used for the database polling.
I tried asking around to see why that sleep was added to the script, but didn't get an answer, so I added it in both sections for safety, I can easily remove if we understand the purpose ;-) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #203890|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 --- Comment #9 from David Nind <david@davidnind.com> --- Created attachment 204069 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=204069&action=edit Bug 36549: Use Parallel::ForkManager in es_indexer_daemon This patch copies the implementation from the general background worked into the es_indexer daemon Note: There is a 'sleep(10)' in the general indexer and I copied it here, but I don't know why we do it? To test: 1 - Apply the patch 2 - sudo koha-es-indexer --restart kohadev 3 - perl misc/maintenance/touch_all_biblios.pl 4 - Wait a bit 5 - Check the background jobs in the admin interface 6 - Confirm the jobs finish and records are searchable 7 - Test with and without rabbitmq enabled: . Stop rabbitmq service: sudo -s service rabbitmq-server stop . Repeat steps 2 to 5 8 - Sign off Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 --- Comment #10 from David Nind <david@davidnind.com> --- (In reply to Nick Clemens (kidclamp) from comment #7)
Did you restart the indexer after stopping rabbit? The errors trying to connect are expected, but new jobs should be done via polling
I have now 8-) - now signed off. (I updated the test plan in the commit message to reflect this.) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lisette@bywatersolutions.co | |m QA Contact|testopia@bugs.koha-communit |aleisha@catalyst.net.nz |y.org | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #204069|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 --- Comment #11 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 204191 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=204191&action=edit Bug 36549: Use Parallel::ForkManager in es_indexer_daemon This patch copies the implementation from the general background worked into the es_indexer daemon Note: There is a 'sleep(10)' in the general indexer and I copied it here, but I don't know why we do it? To test: 1 - Apply the patch 2 - sudo koha-es-indexer --restart kohadev 3 - perl misc/maintenance/touch_all_biblios.pl 4 - Wait a bit 5 - Check the background jobs in the admin interface 6 - Confirm the jobs finish and records are searchable 7 - Test with and without rabbitmq enabled: . Stop rabbitmq service: sudo -s service rabbitmq-server stop . Repeat steps 2 to 5 8 - Sign off Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Aleisha Amohia <aleisha@catalyst.net.nz> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36549 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|normal |enhancement -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org