[Bug 27115] New: Restarting koha-common fails to restart SIP2 server
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27115 Bug ID: 27115 Summary: Restarting koha-common fails to restart SIP2 server Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Packaging Assignee: koha-bugs@lists.koha-community.org Reporter: joonas.kylmala@helsinki.fi QA Contact: testopia@bugs.koha-community.org CC: mtj@kohaaloha.com Sometimes when we run "systemctl restart koha-commmon" the SIP server doesn't get restarted but might just get stopped leading to longer than expected downtime. There is probably some sort of race condition. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27115 Andrew Nugged <nugged@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nugged@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27115 --- Comment #1 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I am not sure if this is still valid, but at least there is no mention of SIP in the output of koha-common restart: sudo service koha-common restart Restarting Koha ILS: koha-commonStopping Plack daemon for kohadev:. Stopping Z39.50/SRU daemon for kohadev:. Stopping Koha worker daemon for kohadev (default):. Stopping Koha worker daemon for kohadev (long_tasks):. Stopping Koha indexing daemon for kohadev:. Successfully stopped all services for instance koha-common:. Starting Plack daemon for kohadev:. Starting Z39.50/SRU daemon for kohadev:. Starting Koha worker daemon for kohadev (default):. Starting Koha worker daemon for kohadev (long_tasks):. Starting Koha indexing daemon for kohadev:. . -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27115 Liz Rea <wizzyrea@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |wizzyrea@gmail.com --- Comment #2 from Liz Rea <wizzyrea@gmail.com> --- I've seen this too, I think it happens when there are a lot of SIP workers/processes and it takes some time for them to kill. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27115 Lari Taskula <lari.taskula@hypernova.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lari.taskula@hypernova.fi --- Comment #3 from Lari Taskula <lari.taskula@hypernova.fi> --- This is still valid. Upping verbosity for koha-sip --start reveals the underlying issue. The SIP daemon doesn't stop in time to be able to start again. During koha-common startup, we are checking if SIP is already running. If so, we will not start the SIP server, and it will stay in stopped state until it is manually started again. See line 11 "SIP server already running for ...":
1 06:07:43.x systemd[1]: Stopping koha-common.service - Start required services for each Koha instance... 2 06:07:43.x koha-common[x]: * Stopping SIP server for production_instance 3 06:07:43.x koha-common[x]: ...done. 4 ... 5 06:07:44.x systemd[1]: koha-common.service: Deactivated successfully. 6 06:07:44.x systemd[1]: koha-common.service: Unit process x (daemon) remains running after unit stopped. 7 06:07:44.x systemd[1]: koha-common.service: Unit process x (es_indexer_daem) remains running after unit stopped. 8 06:07:44.x systemd[1]: Stopped koha-common.service - Start required services for each Koha instance 9 ... 10 06:07:44.x systemd[1]: Starting koha-common.service - Start required services for each Koha instance... 11 06:07:45.x koha-common[x]: * Warning: SIP server already running for production_instance
-- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27115 Lari Taskula <lari.taskula@hypernova.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27115 --- Comment #4 from Lari Taskula <lari.taskula@hypernova.fi> --- Created attachment 188707 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=188707&action=edit Bug 27115: Use start-stop-daemon to stop in koha-sip There has been an issue where a restart of koha-sip would sometimes not bring the SIP server back alive. This is because we have not been waiting for koha-sip --stop to actually stop the SIP server(s). It is then possible for a restart to attempt the --start command while a previous instance of SIP server is still running. As the --start command in this case is not allowed, we would then end up not starting the SIP server(s) at all. This patch replaces daemon --stop with start-stop-daemon --stop to ensure koha-sip daemon actually stops. start-stop-daemon provides us the --retry option, which we are able to utilize in waiting for the stop process to finish. To test: 1. Before applying this patch 2. Run this command export SIPINSTANCE="your_koha_instance_name" You can your_koha_instance_name using the koha-list command 3. Configure a SIP server (add some min_servers, e.g. 6, to SIPConfig.xml) 4. koha-sip --enable $SIPINSTANCE 5. koha-sip --verbose --start $SIPINSTANCE 6. Observe "Starting SIP server for ..." 7. koha-sip --verbose --stop $SIPINSTANCE; ps aux | grep $SIPINSTANCE-koha-sip 8. Observe "Stopping SIP server for instancename" 9. In the output, observe the number of SIP servers you defined in step 3, plus one. 10. koha-sip --verbose --start $SIPINSTANCE 11. Apply this patch, or edit the koha-sip directly to get changes in place in your environment, e.g. using command nano $(which koha-sip) 12. koha-sip --verbose --stop $SIPINSTANCE; ps aux | grep $SIPINSTANCE-koha-sip 13. Observe "Stopping SIP server for instancename" 14. In the output, observe no more SIP servers alive -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27115 --- Comment #5 from Lari Taskula <lari.taskula@hypernova.fi> --- (In reply to Lari Taskula from comment #3)
7 06:07:44.x systemd[1]: koha-common.service: Unit process x (es_indexer_daem) remains running after unit stopped.
Btw this line also concerns me. A couple of months ago we had a case where ES indexer was suddenly no longer running, most likely after a restart. Could be related to this issue and perhaps we should use start-stop-daemon for koha-es-indexer --stop as well. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27115 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27115 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #188707|0 |1 is obsolete| | --- Comment #6 from David Nind <david@davidnind.com> --- Created attachment 190766 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=190766&action=edit Bug 27115: Use start-stop-daemon to stop in koha-sip There has been an issue where a restart of koha-sip would sometimes not bring the SIP server back alive. This is because we have not been waiting for koha-sip --stop to actually stop the SIP server(s). It is then possible for a restart to attempt the --start command while a previous instance of SIP server is still running. As the --start command in this case is not allowed, we would then end up not starting the SIP server(s) at all. This patch replaces daemon --stop with start-stop-daemon --stop to ensure koha-sip daemon actually stops. start-stop-daemon provides us the --retry option, which we are able to utilize in waiting for the stop process to finish. To test: 1. Before applying this patch 2. Run this command export SIPINSTANCE="your_koha_instance_name" You can your_koha_instance_name using the koha-list command 3. Configure a SIP server (add some min_servers, e.g. 6, to SIPConfig.xml) 4. koha-sip --enable $SIPINSTANCE 5. koha-sip --verbose --start $SIPINSTANCE 6. Observe "Starting SIP server for ..." 7. koha-sip --verbose --stop $SIPINSTANCE; ps aux | grep $SIPINSTANCE-koha-sip 8. Observe "Stopping SIP server for instancename" 9. In the output, observe the number of SIP servers you defined in step 3, plus one. 10. koha-sip --verbose --start $SIPINSTANCE 11. Apply this patch, or edit the koha-sip directly to get changes in place in your environment, e.g. using command nano $(which koha-sip) 12. koha-sip --verbose --stop $SIPINSTANCE; ps aux | grep $SIPINSTANCE-koha-sip 13. Observe "Stopping SIP server for instancename" 14. In the output, observe no more SIP servers alive Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27115 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the| |This fixes an issue when release notes| |stopping and restarting SIP | |servers using `koha-sip` - | |it would sometimes not | |restart the SIP servers. | | | |This was because a restart | |could attempt the --start | |command while a previous | |SIP server was still | |running. This could result | |in the SIP server not | |restarting at all. | | | |The fix replaces `daemon | |--stop` with | |`start-stop-daemon --stop` | |in the code, to ensure that | |all the running SIP servers | |are actually stopped before | |restarting. CC| |david@davidnind.com --- Comment #7 from David Nind <david@davidnind.com> --- I think I tested this correctly, and that I observed the expected results! In KTD, the SIP server is already running, but I went through the process of stopping it and then disabling it before working through the test plan. I've attempted a release note. Testing notes (using KTD): 1. SIP is already running in KTD for kohadev: sudo koha-sip --verbose --status kohadev SIP server running for kohadev:. 2. Make the change in step 3 of the test plan - made min_servers = 6: vi /etc/koha/sites/kohadev/SIPconfig.xml 3. Stop the SIP server: sudo koha-sip --verbose --stop kohadev Stopping SIP server for kohadev:. 4. Disable the SIP server: sudo koha-sip --verbose --disable kohadev Stopping running SIP Warning: SIP server not running for kohadev:. Information: SIP server disabled for kohadev 5. Check that no SIP servers are running: ps aux | grep kohadev-koha-sip kohadev+ 17009 0.0 0.0 9008 2176 pts/0 S+ 18:07 0:00 grep kohadev-koha-sip sudo koha-sip --verbose --status kohadev SIP server not running for kohadev: failed! 6. Enable the SIP server: sudo koha-sip --verbose --enable kohadev Enabling SIP server for kohadev - edit /etc/koha/sites/kohadev/SIPconfig.xml to configure 7. Start the SIP server: sudo koha-sip --verbose --start kohadev Starting SIP server for kohadev:. 8. Check the number of servers running - this is 7 for me (min_servers = 6): ps aux | grep $SIPINSTANCE-koha-sip kohadev+ 17366 0.0 0.0 8176 1808 ? S 18:12 0:00 daemon --name=kohadev-koha-sip --errlog=/var/log/koha/kohadev/sip-error.log --output=/var/log/koha/kohadev/sip-output.log --verbose=1 --respawn --delay=30 --pidfiles=/var/run/koha/kohadev --user=kohadev-koha.kohadev-koha -- perl /kohadevbox/koha/C4/SIP/SIPServer.pm /etc/koha/sites/kohadev/SIPconfig.xml kohadev+ 17368 0.8 0.6 308364 208892 ? S 18:12 0:01 kohadev-koha-sip: perl /kohadevbox/koha/C4/SIP/SIPServer.pm /etc/koha/sites/kohadev/SIPconfig.xml kohadev+ 17373 0.0 0.5 308364 187492 ? S 18:12 0:00 kohadev-koha-sip: perl /kohadevbox/koha/C4/SIP/SIPServer.pm /etc/koha/sites/kohadev/SIPconfig.xml kohadev+ 17374 0.0 0.5 308364 187492 ? S 18:12 0:00 kohadev-koha-sip: perl /kohadevbox/koha/C4/SIP/SIPServer.pm /etc/koha/sites/kohadev/SIPconfig.xml kohadev+ 17375 0.0 0.5 308364 187492 ? S 18:12 0:00 kohadev-koha-sip: perl /kohadevbox/koha/C4/SIP/SIPServer.pm /etc/koha/sites/kohadev/SIPconfig.xml kohadev+ 17376 0.0 0.5 308364 187492 ? S 18:12 0:00 kohadev-koha-sip: perl /kohadevbox/koha/C4/SIP/SIPServer.pm /etc/koha/sites/kohadev/SIPconfig.xml kohadev+ 17377 0.0 0.5 308364 187492 ? S 18:12 0:00 kohadev-koha-sip: perl /kohadevbox/koha/C4/SIP/SIPServer.pm /etc/koha/sites/kohadev/SIPconfig.xml kohadev+ 17378 0.0 0.5 308364 187492 ? S 18:12 0:00 kohadev-koha-sip: perl /kohadevbox/koha/C4/SIP/SIPServer.pm /etc/koha/sites/kohadev/SIPconfig.xml kohadev+ 17517 0.0 0.0 9008 2304 pts/0 S+ 18:14 0:00 grep kohadev-koha-sip 9. Back to step 7 of the test plan: sudo koha-sip --verbose --stop kohadev; ps aux | grep kohadev-koha-sip Stopping SIP server for kohadev:. kohadev+ 17366 0.0 0.0 8176 1808 ? S 18:12 0:00 daemon --name=kohadev-koha-sip --errlog=/var/log/koha/kohadev/sip-error.log --output=/var/log/koha/kohadev/sip-output.log --verbose=1 --respawn --delay=30 --pidfiles=/var/run/koha/kohadev --user=kohadev-koha.kohadev-koha -- perl /kohadevbox/koha/C4/SIP/SIPServer.pm /etc/koha/sites/kohadev/SIPconfig.xml kohadev+ 17368 0.4 0.6 308364 208892 ? R 18:12 0:01 kohadev-koha-sip: perl /kohadevbox/koha/C4/SIP/SIPServer.pm /etc/koha/sites/kohadev/SIPconfig.xml kohadev+ 17373 0.0 0.5 308364 188900 ? R 18:12 0:00 kohadev-koha-sip: perl /kohadevbox/koha/C4/SIP/SIPServer.pm /etc/koha/sites/kohadev/SIPconfig.xml kohadev+ 17374 0.0 0.5 308364 188900 ? R 18:12 0:00 kohadev-koha-sip: perl /kohadevbox/koha/C4/SIP/SIPServer.pm /etc/koha/sites/kohadev/SIPconfig.xml kohadev+ 17375 0.0 0.5 308364 188900 ? R 18:12 0:00 kohadev-koha-sip: perl /kohadevbox/koha/C4/SIP/SIPServer.pm /etc/koha/sites/kohadev/SIPconfig.xml kohadev+ 17376 0.0 0.5 308364 188900 ? R 18:12 0:00 kohadev-koha-sip: perl /kohadevbox/koha/C4/SIP/SIPServer.pm /etc/koha/sites/kohadev/SIPconfig.xml kohadev+ 17377 0.0 0.5 308364 188900 ? R 18:12 0:00 kohadev-koha-sip: perl /kohadevbox/koha/C4/SIP/SIPServer.pm /etc/koha/sites/kohadev/SIPconfig.xml kohadev+ 17378 0.0 0.5 308364 188900 ? R 18:12 0:00 kohadev-koha-sip: perl /kohadevbox/koha/C4/SIP/SIPServer.pm /etc/koha/sites/kohadev/SIPconfig.xml kohadev+ 17666 0.0 0.0 9008 2048 pts/0 S+ 18:16 0:00 grep kohadev-koha-sip 10. Step 10 of the test plan - start the SIP server (7 running after this): sudo koha-sip --verbose --start kohadev Starting SIP server for kohadev:. 11. After applying the patch - end result is 7 SIP servers running (from e ps aux command): reset_all + step 2 above + restart_all 12. Steps 12 to 14 of the test plan: sudo koha-sip --verbose --stop kohadev; ps aux | grep kohadev-koha-sip Stopping SIP server for kohadev:. kohadev+ 20224 0.0 0.0 9008 2048 pts/0 S+ 18:28 0:00 grep kohadev-koha-sip -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27115 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au --- Comment #8 from David Cook <dcook@prosentient.com.au> --- I have noticed that there have been times where the SIP server doesn't seem to come back up after a restart... Can't reproduce it on KTD though. I'm not sure if we want to just use start-stop-daemon for stop in koha-sip. That said, there appears to be precedence for it in "debian/scripts/koha-z3950-responder" and "debian/scripts/koha-plack" so maybe it is fine after all actually. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27115 --- Comment #9 from David Cook <dcook@prosentient.com.au> --- The test plan for this one is bad though. It's using the built in /sbin/koha-sip instead of the one at debian/scripts/koha-sip. You need to copy debian/scripts/koha-sip to /sbin/koha-sip to test this fully. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27115 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27115 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #190766|0 |1 is obsolete| | --- Comment #10 from David Cook <dcook@prosentient.com.au> --- Created attachment 191546 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=191546&action=edit Bug 27115: Use start-stop-daemon to stop in koha-sip There has been an issue where a restart of koha-sip would sometimes not bring the SIP server back alive. This is because we have not been waiting for koha-sip --stop to actually stop the SIP server(s). It is then possible for a restart to attempt the --start command while a previous instance of SIP server is still running. As the --start command in this case is not allowed, we would then end up not starting the SIP server(s) at all. This patch replaces daemon --stop with start-stop-daemon --stop to ensure koha-sip daemon actually stops. start-stop-daemon provides us the --retry option, which we are able to utilize in waiting for the stop process to finish. To test: 1. Before applying this patch 2. Run this command export SIPINSTANCE="your_koha_instance_name" You can your_koha_instance_name using the koha-list command 3. Configure a SIP server (add some min_servers, e.g. 6, to SIPConfig.xml) 4. koha-sip --enable $SIPINSTANCE 5. koha-sip --verbose --start $SIPINSTANCE 6. Observe "Starting SIP server for ..." 7. koha-sip --verbose --stop $SIPINSTANCE; ps aux | grep $SIPINSTANCE-koha-sip 8. Observe "Stopping SIP server for instancename" 9. In the output, observe the number of SIP servers you defined in step 3, plus one. 10. koha-sip --verbose --start $SIPINSTANCE 11. Apply this patch, or edit the koha-sip directly to get changes in place in your environment, e.g. using command nano $(which koha-sip) 12. koha-sip --verbose --stop $SIPINSTANCE; ps aux | grep $SIPINSTANCE-koha-sip 13. Observe "Stopping SIP server for instancename" 14. In the output, observe no more SIP servers alive Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: David Cook <dcook@prosentient.com.au> -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27115 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |dcook@prosentient.com.au |y.org | Assignee|koha-bugs@lists.koha-commun |lari.taskula@hypernova.fi |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27115 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Component|Packaging |Architecture, internals, | |and plumbing -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27115 Lucas Gass (lukeg) <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to main Version(s)| |26.05.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27115 --- Comment #11 from Lucas Gass (lukeg) <lucas@bywatersolutions.com> --- Nice work everyone! Pushed to main for 26.05 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27115 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=27140 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27115 --- Comment #12 from David Cook <dcook@prosentient.com.au> --- Looking at bug 27140 and realising that I thought of the start-stop-daemon ages ago but never applied it to koha-zebra. Oops... -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27115 Jacob O'Mara <jacob.omara@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|26.05.00 |26.05.00,25.11.02 released in| | Status|Pushed to main |Pushed to stable -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27115 --- Comment #13 from Jacob O'Mara <jacob.omara@openfifth.co.uk> --- Thanks all, pushed to 25.11.x -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27115 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m Resolution|--- |FIXED Status|Pushed to stable |RESOLVED --- Comment #14 from Fridolin Somers <fridolin.somers@biblibre.com> --- Looks ricky to impact older versions on this major script -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org