[Bug 16885] New: koha-stop-zebra should be more sure of stopping zebrasrv
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16885 Bug ID: 16885 Summary: koha-stop-zebra should be more sure of stopping zebrasrv Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Packaging Assignee: koha-bugs@lists.koha-community.org Reporter: gmcharlt@gmail.com QA Contact: testopia@bugs.koha-community.org CC: mirko@abunchofthings.net The koha-stop-zebra command can finish without ensuring that zebrasrv is actually stopped. In particular, if there is an active query running when daemon --stop is run, the following state of affairs will occur until the query finishes or times out: - the child zebrasrv processing the query continues to run - the parent zebrasrv process becomes a zombie - the daemon(1) process continues to run -- and doesn't delete the lock file until the children go away. This is particularly problematic when a koha-stop-zebra is followed by a koha-start-zebra in quick succession, as happens during a logrotate. This is because while zebrasrv is in the half-stopped, half-started stop, the lock file still exists... and koha-start-zebra will see it and believe that zebrasrv is still running. When the query holding the child zebrasrv process finishes, the instance is left with no running Zebra. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16885 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |m.de.rooy@rijksmuseum.nl -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16885 --- Comment #1 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- I think this explains a case where I had a stopped zebrasrv on the day the logs rotated. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16885 M. Tompsett <mtompset@hotmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |manueltamayao@gmail.com, | |mtompset@hotmail.com --- Comment #2 from M. Tompsett <mtompset@hotmail.com> --- Sounds exactly like what we are experiencing. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16885 Mason James <mtj@kohaaloha.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mtj@kohaaloha.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16885 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |philippe.blouin@inlibro.com --- Comment #3 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> ---
From Philippe Blouin on the dev ml: === QUOTE We've had this problem on 10% of our installations. I solved it by going nuclear with logrotate since that was really hurting the users to not have zebra working (it would appear like working on our monitor, but was actually just dying and restarting).
Here is the postrotate I've end up with, after many iterations of Next Level Apocalypse You :) postrotate service apache2 reload find /etc/init.d/ -name "*zebra-daemon" -exec bash -c 'service "$(basename "$0")" stop' {} \; sleep 3 find /etc/init.d/ -name "*zebra-daemon" -exec bash -c 'service "$(basename "$0")" stop' {} \; sleep 3 pkill -9 zebra sleep 2 pkill -9 zebra sleep 2 find /etc/init.d/ -name "*zebra-daemon" -exec bash -c 'service "$(basename "$0")" start' {} \; endscript As you see, no subtlety, but I've had no problem for a year now... == END QUOTE -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16885 M. Tompsett <mtompset@hotmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff Patch complexity|--- |Small patch -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16885 --- Comment #4 from M. Tompsett <mtompset@hotmail.com> --- Created attachment 53925 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=53925&action=edit Bug 16885: koha-stop-zebra should be more sure of stopping zebrasrv This patch actually loops 10 tens to check, before giving up. TEST PLAN --------- 1) Back up your koha logs as desired. 2) add something to /var/log/koha/{instance name}/intranet-error.log 3) ps aux | grep zebra 4) logrotate -f /etc/logrotate.d/koha-common 5) ps aux | grep zebra -- the zebrasrv and daemon process for zebra indexing didn't restart. 6) apply this patch against /usr/sbin/koha-stop-zebra 7) sudo koha-start-zebra {instance name} 8) ps aux | grep zebra -- the processes should have started up again. 9) add different junk to /var/log/koha/{instance name}/intranet-error.log 10) ps aux | grep zebra 11) logrotate -f /etc/logrotate.d/koha-common 12) ps aux | grep zebra -- the process ids for the zebrasrv and daemon processes should be different, but the number of processes is the same as before. 13) sign off, because its less ugly than comment #3 -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16885 M. Tompsett <mtompset@hotmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #53925|0 |1 is obsolete| | --- Comment #5 from M. Tompsett <mtompset@hotmail.com> --- Created attachment 53926 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=53926&action=edit Bug 16885: koha-stop-zebra should be more sure of stopping zebrasrv This patch actually loops 10 tens to check, before giving up. TEST PLAN --------- 1) Back up your koha logs as desired. 2) add something to /var/log/koha/{instance name}/intranet-error.log 3) ps aux | grep zebra 4) logrotate -f /etc/logrotate.d/koha-common 5) ps aux | grep zebra -- the zebrasrv and daemon process for zebra indexing didn't restart. 6) apply this patch against /usr/sbin/koha-stop-zebra 7) sudo koha-start-zebra {instance name} 8) ps aux | grep zebra -- the processes should have started up again. 9) add different junk to /var/log/koha/{instance name}/intranet-error.log 10) ps aux | grep zebra 11) logrotate -f /etc/logrotate.d/koha-common 12) ps aux | grep zebra -- the process ids for the zebrasrv and daemon processes should be different, but the number of processes is the same as before. 13) sign off, because its less ugly than comment #3 Sponsored-by: Tulong Aklatan -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16885 J Schmidt <jschmidt@switchinc.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jschmidt@switchinc.org -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16885 --- Comment #6 from Galen Charlton <gmcharlt@gmail.com> --- Comment on attachment 53926 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=53926 Bug 16885: koha-stop-zebra should be more sure of stopping zebrasrv Review of attachment 53926: --> (https://bugs.koha-community.org/bugzilla3/page.cgi?id=splinter.html&bug=16885&attachment=53926) ----------------------------------------------------------------- ::: debian/scripts/koha-stop-zebra @@ +35,5 @@
+ # both daemon and zebrasrv processes for one instance + # will go away, but there could be other processes + # for other instances. + expecting=`ps aux | grep zebra | wc -l` + expecting=$((expecting-2))
zebrasrv forks a process for every active connection, so one can't count on there being exactly two processes. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16885 --- Comment #7 from Galen Charlton <gmcharlt@gmail.com> --- Per my previous comment, I don't think Mark's patch will be reliable. I think a better approach would be to use start-stop-daemon, a la the init script that ships with IndexData's own package of Zebra (https://github.com/indexdata/idzebra/blob/master/debian/idzebra-2.0-utils.ze...): start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE --name $NAME Unfortunately, daemon(1) doesn't appear to have any ability to specify using SIGKILL. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16885 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@bugs.koha-c | |ommunity.org, | |tomascohen@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16885 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA --- Comment #8 from Tomás Cohen Arazi <tomascohen@gmail.com> --- (In reply to Galen Charlton from comment #7)
Per my previous comment, I don't think Mark's patch will be reliable.
I think a better approach would be to use start-stop-daemon, a la the init script that ships with IndexData's own package of Zebra (https://github.com/indexdata/idzebra/blob/master/debian/idzebra-2.0-utils. zebrasrv.init#L69):
start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE --name $NAME
Unfortunately, daemon(1) doesn't appear to have any ability to specify using SIGKILL.
I agree, and we already use start-stop-daemon koha-plack. Good catch, Galen! -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16885 M. Tompsett <mtompset@hotmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16885 M. Tompsett <mtompset@hotmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #53926|0 |1 is obsolete| | --- Comment #9 from M. Tompsett <mtompset@hotmail.com> --- Created attachment 53958 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=53958&action=edit Bug 16885: koha-stop-zebra should be more sure of stopping zebrasrv This patch follows Galen's suggestion in comment #7. TEST PLAN --------- 1) Back up your koha logs as desired. 2) add something to /var/log/koha/{instance name}/intranet-error.log 3) ps aux | grep zebra 4) logrotate -f /etc/logrotate.d/koha-common 5) ps aux | grep zebra -- the zebrasrv and daemon process for zebra indexing didn't restart. 6) apply this patch against /usr/sbin/koha-stop-zebra 7) sudo koha-start-zebra {instance name} 8) ps aux | grep zebra -- the processes should have started up again. 9) add different junk to /var/log/koha/{instance name}/intranet-error.log 10) ps aux | grep zebra 11) logrotate -f /etc/logrotate.d/koha-common 12) ps aux | grep zebra -- the process ids for the zebrasrv and daemon processes should be different, but the number of processes is the same as before. 13) sign off, because its less ugly than comment #3 Sponsored-by: Tulong Aklatan -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16885 M. Tompsett <mtompset@hotmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |mtompset@hotmail.com |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16885 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16885 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #53958|0 |1 is obsolete| | --- Comment #10 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 54314 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=54314&action=edit Bug 16885: koha-stop-zebra should be more sure of stopping zebrasrv This patch follows Galen's suggestion in comment #7. TEST PLAN --------- 1) Back up your koha logs as desired. 2) add something to /var/log/koha/{instance name}/intranet-error.log 3) ps aux | grep zebra 4) logrotate -f /etc/logrotate.d/koha-common 5) ps aux | grep zebra -- the zebrasrv and daemon process for zebra indexing didn't restart. 6) apply this patch against /usr/sbin/koha-stop-zebra 7) sudo koha-start-zebra {instance name} 8) ps aux | grep zebra -- the processes should have started up again. 9) add different junk to /var/log/koha/{instance name}/intranet-error.log 10) ps aux | grep zebra 11) logrotate -f /etc/logrotate.d/koha-common 12) ps aux | grep zebra -- the process ids for the zebrasrv and daemon processes should be different, but the number of processes is the same as before. 13) sign off, because its less ugly than comment #3 Sponsored-by: Tulong Aklatan Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Works as expected, no regressions found. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16885 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16885 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #54314|0 |1 is obsolete| | --- Comment #11 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 54373 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=54373&action=edit Bug 16885: koha-stop-zebra should be more sure of stopping zebrasrv This patch follows Galen's suggestion in comment #7. TEST PLAN --------- 1) Back up your koha logs as desired. 2) add something to /var/log/koha/{instance name}/intranet-error.log 3) ps aux | grep zebra 4) logrotate -f /etc/logrotate.d/koha-common 5) ps aux | grep zebra -- the zebrasrv and daemon process for zebra indexing didn't restart. 6) apply this patch against /usr/sbin/koha-stop-zebra 7) sudo koha-start-zebra {instance name} 8) ps aux | grep zebra -- the processes should have started up again. 9) add different junk to /var/log/koha/{instance name}/intranet-error.log 10) ps aux | grep zebra 11) logrotate -f /etc/logrotate.d/koha-common 12) ps aux | grep zebra -- the process ids for the zebrasrv and daemon processes should be different, but the number of processes is the same as before. 13) sign off, because its less ugly than comment #3 Sponsored-by: Tulong Aklatan Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Works as expected, no regressions found. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16885 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to Master CC| |kyle@bywatersolutions.com --- Comment #12 from Kyle M Hall <kyle@bywatersolutions.com> --- Pushed to master for 16.11, thanks Mark! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16885 Frédéric Demians <frederic@tamil.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to Master |Pushed to Stable CC| |frederic@tamil.fr --- Comment #13 from Frédéric Demians <frederic@tamil.fr> --- Pushed in 16.05. Will be in 16.05.03. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16885 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |julian.maurice@biblibre.com --- Comment #14 from Julian Maurice <julian.maurice@biblibre.com> --- Pushed to 3.22.x, will be in 3.22.10 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16885 Marc Véron <veron@veron.ch> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=17814 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16885 Natsagdorj <natsagdorj@msue.edu.mn> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |natsagdorj@msue.edu.mn -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org