[Bug 15006] New: Need to distinguish client timeout from login timeout
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15006 Bug ID: 15006 Summary: Need to distinguish client timeout from login timeout Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: SIP2 Assignee: koha-bugs@lists.koha-community.org Reporter: colin.campbell@ptfs-europe.com QA Contact: testopia@bugs.koha-community.org CC: colin.campbell@ptfs-europe.com Since patch 13432 was applied the main sip loop times out according to the timeout value. This can cause considerable instability and cause client machines to shutdown if the value is too low. As the timeout variable is a login timeout it is typically on the low side (30sec) for client connections it needs to be over 5mins typically possibly more. Suggest a the time out for active connections be separated from the login timeout. we also need the possibility to switch it off completely. Submit a patch adding a 'client_timeout' parameter -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15006 Magnus Enger <magnus@libriotech.no> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |magnus@libriotech.no -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15006 Colin Campbell <colin.campbell@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |colin.campbell@ptfs-europe. |ity.org |com Status|NEW |Needs Signoff --- Comment #1 from Colin Campbell <colin.campbell@ptfs-europe.com> --- Created attachment 45039 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=45039&action=edit Proposed Patch Testing - without patch an active sip connection will be timed out after timeout seconds without receiving a request from the client. With patch applied connection will remain open until client closed. Time out can be reinstated by adding a client_timeout parameter as per the sample config, the separate name allows this to be defined at a different duration from the login timeout -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15006 Frédéric Demians <frederic@tamil.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |frederic@tamil.fr --- Comment #2 from Frédéric Demians <frederic@tamil.fr> --- With this: my $timeout = $self->{service}->{client_timeout} || $config->{client_timeout}; what will happen after Koha upgrade if a library doesn't add the new 'client_timeout' parameter to its config file. Wouldn't it be better to have something like: my $timeout = $self->{service}->{client_timeout} || $config->{client_timeout} || $self->{service}->{timeout} || $config->{timeout}; And/or a note via updatabase.pl to notify that SIP config file has to be reviewed after upgrade. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15006 --- Comment #3 from Colin Campbell <colin.campbell@ptfs-europe.com> --- (In reply to Frédéric Demians from comment #2)
With this:
my $timeout = $self->{service}->{client_timeout} || $config->{client_timeout};
what will happen after Koha upgrade if a library doesn't add the new 'client_timeout' parameter to its config file. Wouldn't it be better to have something like:
my $timeout = $self->{service}->{client_timeout} || $config->{client_timeout} || $self->{service}->{timeout} || $config->{timeout};
And/or a note via updatabase.pl to notify that SIP config file has to be reviewed after upgrade.
If the client_timeout is not present the current situation remains the same (or in fact reverts to the situation before timeout was incorrectrly extended to timeout active connections) Your suggestions for adding it to the test preserves the bad code from patch 13432 which has caused in my experience to make previously working installs go unstable. Once this gives a working system, I'm hoping to overhaul the cxonfig file which is under documented -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15006 Frédéric Demians <frederic@tamil.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15006 Frédéric Demians <frederic@tamil.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #45039|0 |1 is obsolete| | --- Comment #4 from Frédéric Demians <frederic@tamil.fr> --- Created attachment 45583 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=45583&action=edit Bug 15006 Sip separate global from login timeouts Timeout logic has been added to the main sip loop However the timeout parameter was initially for the login process which requires a short duration applying such a brief timeout to the main sip loop causes instability in many selfcheck units Most selfchecks have persistent connections and send a periodic status request at intervals (approx every 5mins appears the norm) The timeout was dropping connections by default every 30secs which for the client appears as a very flakey network This patch adds a separate parameter client_timeout that can be used if you do want to force a disconnect if the client sends no requests for a period. By not defining it you can switch off the timeout in the main loop, I recommend this unless you have a good reason not too. If you do want this behaviour you can at least now set it to a more realistic value (see sample config file) Removed the commented out html entry in the config file, this was dead code born of a misconception and was just a source of confusion for users Signed-off-by: Frédéric Demians <f.demians@tamil.fr> -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15006 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15006 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #45583|0 |1 is obsolete| | --- Comment #5 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 45594 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=45594&action=edit Bug 15006 Sip separate global from login timeouts Timeout logic has been added to the main sip loop However the timeout parameter was initially for the login process which requires a short duration applying such a brief timeout to the main sip loop causes instability in many selfcheck units Most selfchecks have persistent connections and send a periodic status request at intervals (approx every 5mins appears the norm) The timeout was dropping connections by default every 30secs which for the client appears as a very flakey network This patch adds a separate parameter client_timeout that can be used if you do want to force a disconnect if the client sends no requests for a period. By not defining it you can switch off the timeout in the main loop, I recommend this unless you have a good reason not too. If you do want this behaviour you can at least now set it to a more realistic value (see sample config file) Removed the commented out html entry in the config file, this was dead code born of a misconception and was just a source of confusion for users Signed-off-by: Frédéric Demians <f.demians@tamil.fr> Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15006 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |13432 CC| |jonathan.druart@bugs.koha-c | |ommunity.org Severity|enhancement |normal Referenced Bugs: http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13432 [Bug 13432] SIP Server does not respect timeout setting -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15006 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |kyle@bywatersolutions.com Status|Passed QA |Failed QA --- Comment #6 from Kyle M Hall <kyle@bywatersolutions.com> ---
If the client_timeout is not present the current situation remains the same (or in fact reverts to the situation before timeout was incorrectrly extended to timeout active connections) Your suggestions for adding it to the test preserves the bad code from patch 13432 which has caused in my experience to make previously working installs go unstable. Once this gives a working system, I'm hoping to overhaul the cxonfig file which is under documented
Bug 13432 patched a serious problem where child sip process would never die and thus the SIP server would continue to take up more and more memory until a restart was required to kill all the child processes. I'm fairly certain this patch would bring back that situation. I would like to see a default timeout in the code instead of allowing client_timeout to not be set. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15006 Colin Campbell <colin.campbell@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Signed Off --- Comment #7 from Colin Campbell <colin.campbell@ptfs-europe.com> --- Bug 13432 does not fix the bug you describe. There is (hard to reproduce in testing but found in practice) bug which is caused by a child failing to receive the kill signal from the parent, or possibly looping on an operation subsequent to a call. Problem arises from the way the protocol loop is coded This patch will still allow you to timeout a call should you wish but also negate the necessity of reverting 13432 in production because it causes units to fail due to the server timing out valid connections -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15006 --- Comment #8 from Kyle M Hall <kyle@bywatersolutions.com> --- Thanks for the explanation! I'm going to ask Larry to lend some eyes to this. I'm just want to make sure we don't introduce a regression. I think another alternative would be to set the client_timeout to default to the timeout if client_timeout is not defined. You could then use a client_timeout of 0 to specify no timeout at all. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15006 Larry Baerveldt <larry@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |larry@bywatersolutions.com --- Comment #9 from Larry Baerveldt <larry@bywatersolutions.com> --- Like Kyle, I am concerned about regression. The issue addressed by Bug 13432 was a very serious one for us, and affected virtually every site using SIP. New child processes were continually being created throughout the day, and older child processes were never dying. In some cases the number of SIP processes would literally max out the RAM. We had to introduce daily restarts of SIP in our crons, and for some sites, even that was not enough. And Bug 13432 DID fix that problem. As to the problem of the timeout logic causing dropped connections post 13432, we simply fixed that on our servers by increasing the timeout in the Service and Institution levels to 360. That worked for most SIP vendors. That said, I think this patch would be okay if it includes Kyle's suggestion to have the client_timeout default to the timeout if not defined. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15006 --- Comment #10 from Colin Campbell <colin.campbell@ptfs-europe.com> --- (In reply to Larry Baerveldt from comment #9)
Like Kyle, I am concerned about regression. The issue addressed by Bug 13432 was a very serious one for us, and affected virtually every site using SIP. New child processes were continually being created throughout the day, and older child processes were never dying. In some cases the number of SIP processes would literally max out the RAM. We had to introduce daily restarts of SIP in our crons, and for some sites, even that was not enough. And Bug 13432 DID fix that problem.
As to the problem of the timeout logic causing dropped connections post 13432, we simply fixed that on our servers by increasing the timeout in the Service and Institution levels to 360. That worked for most SIP vendors.
That said, I think this patch would be okay if it includes Kyle's suggestion to have the client_timeout default to the timeout if not defined.
If you apply the change in bug 15418 do you still see this problem? In Pre-Fork child processes should not die - but my suspicion is that due to the rather awkward way the callback was originally implemented they are not always left in a state ready to be reused. In part to deal with those problems I started using Fork in production and it seems more reliable -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15006 --- Comment #11 from Larry Baerveldt <larry@bywatersolutions.com> --- We haven't seen the problem (of outrageous growth in number of SIP processes) since Bug 13432 was applied, so there's no way to see if 15418 helps any with that or not. As noted, Bug 13432 solved a very huge problem for us, and any issues with SIP devices timing out was easily solved by adjusting the timeout settings in SIPconfig.xml. We just don't want to see anything that may regress that. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15006 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |katrin.fischer@bsz-bw.de --- Comment #12 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I am not sure I fully understand the concerns here. Of course we should take care not to introduce regressions, but wouldn't it be worth taking a deeper look at the problem and seeing if we can improve stability even more? -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15006 --- Comment #13 from Kyle M Hall <kyle@bywatersolutions.com> --- I will try to find some time to run some tests. Here is what I will do: 1) Revert 13432 on master 2) Set up a script that continuously makes sip2 calls via sip_cli_emulator.pl 3) Let that run for a long time 4) Note whether memory usage increases dramatically 5) Apply this patch 6) Repeat steps 2-4 6) Apply bug 15006 7) Repeat steps 2-4 If everything checks out, we should move ahead with this. Colin, does that seem like a reasonable test plan? -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon@bugs.koha-community.org