[Bug 37087] New: Add support for TCP keepalive to SIP server
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37087 Bug ID: 37087 Summary: Add support for TCP keepalive to SIP server Change sponsored?: --- Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: SIP2 Assignee: koha-bugs@lists.koha-community.org Reporter: dcook@prosentient.com.au QA Contact: testopia@bugs.koha-community.org Some computing platforms like Microsoft Azure (and some corporate firewalls) enforce idle timeouts which silently kill long-running/persistent TCP connections. In the case of Microsoft Azure, this idle timeout occurs after 4 minutes. In production, this has been massively disruptive to circulation operations using self-checkout kiosks that connect to Koha using SIP2. Fortunately, this problem is easily solvable by using TCP keepalives. When TCP keepalives are used, after X seconds a TCP ACK message will be sent from the SIP server to the SIP client, which will then send back a TCP ACK. Nothing happens at the application level. It's just at the lower network level. And this prevents idle timeouts. I'll provide more technical information in code comments. -- I've already successfully implemented this strategy in Azure for managing persistent Elasticsearch TCP connections simply by tweaking the "net.ipv4.tcp_keepalive_time" kernel parameter, which is automatically used by the HTTP::Tiny connection library used by Search::Elasticsearch, since it implements the SO_KEEPALIVE socket option out of the box by default. With the SIP server, we have a lot more control than we do with third-party software, so I'm planning to implement a more fine-grained approach, which won't require any kernel parameter changes. Instead, it will be manageable by application configuration alone. (Note: this is partially because testing in koha-testing-docker would be difficult where it's not easy/possible to tweak kernel parameters.) -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37087 --- Comment #1 from David Cook <dcook@prosentient.com.au> --- I've already written and tested the code for a MVP and it's working as expected. Now it's just about writing the code in a way that is appropriate for the Koha community. I hope to have this ready very shortly. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37087 --- Comment #2 from David Cook <dcook@prosentient.com.au> --- (FYI: note that Net::Stomp and Net::LDAP also have out-of-the-box support for SO_KEEPALIVE.) -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37087 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |dcook@prosentient.com.au |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37087 --- Comment #3 from David Cook <dcook@prosentient.com.au> --- FYI: Keep in mind that TCP keepalive is different but related to HTTP keepalive. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37087 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37087 --- Comment #4 from David Cook <dcook@prosentient.com.au> --- Created attachment 167706 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=167706&action=edit Bug 37087: Add TCP keepalive support to SIP server This change adds the ability to enable and configure TCP keepalive support for the SIP server using SIPconfig.xml. For the sake of backwards compatibility, it defaults to disabled and additional parameters default match typical kernel defaults. Technical detail can be found in the perldoc for C4/SIP/SIPserver.pm Test plan: 0. Apply the patch 1. koha-sip --restart kohadev 2. apt-get update && apt-get install tcpdump 3. In one window, run "tcpdump -A -n -v -i any 'port 6001'" 4. In another window, run the following: echo -e "9300CNterm1|COterm1|CPCPL|\r" | nc 127.0.0.1 6001 -v 5. Note in tcpdump output that after the initial flood of packets, nothing more is received 6. vi /etc/koha/sites/kohadev/SIPconfig.xml 7. In the "server-params" element, add attributes like the following: custom_tcp_keepalive='1' custom_tcp_keepalive_time='10' custom_tcp_keepalive_intvl='5' 8. koha-sip --restart kohadev 9. In one window, run "tcpdump -A -n -v -i any 'port 6001'" 10. In another window, run the following: echo -e "9300CNterm1|COterm1|CPCPL|\r" | nc 127.0.0.1 6001 -v 11. Note in tcpdump output that after the initial flood of packets, ACK packets are sent out every 10+ seconds for the idle connection -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37087 --- Comment #5 from David Cook <dcook@prosentient.com.au> --- If anyone has any questions, I've researched this extensively, including reviewing the Linux kernel. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37087 --- Comment #6 from David Cook <dcook@prosentient.com.au> --- I'm planning to implement this patch in prod next week, as we need the fix ASAP. I've implemented it in the way that I think is best/cleanest for the SIP server. I can understand if people would prefer a per-listener approach for maximum flexibility. However, if folk wanted to go that route, I'd suggest refactoring how we currently handle client_timeout, since post_accept_hook would be a better place than process_request to fetch/set that config. If people don't have strong feelings, we can also always just do it this way, and refactor later if necessary... -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37087 HKS3 Tadeusz Sośnierz <tadeusz@sosnierz.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #167706|0 |1 is obsolete| | --- Comment #7 from HKS3 Tadeusz Sośnierz <tadeusz@sosnierz.com> --- Created attachment 167713 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=167713&action=edit Bug 37087: Add TCP keepalive support to SIP server This change adds the ability to enable and configure TCP keepalive support for the SIP server using SIPconfig.xml. For the sake of backwards compatibility, it defaults to disabled and additional parameters default match typical kernel defaults. Technical detail can be found in the perldoc for C4/SIP/SIPserver.pm Test plan: 0. Apply the patch 1. koha-sip --restart kohadev 2. apt-get update && apt-get install tcpdump 3. In one window, run "tcpdump -A -n -v -i any 'port 6001'" 4. In another window, run the following: echo -e "9300CNterm1|COterm1|CPCPL|\r" | nc 127.0.0.1 6001 -v 5. Note in tcpdump output that after the initial flood of packets, nothing more is received 6. vi /etc/koha/sites/kohadev/SIPconfig.xml 7. In the "server-params" element, add attributes like the following: custom_tcp_keepalive='1' custom_tcp_keepalive_time='10' custom_tcp_keepalive_intvl='5' 8. koha-sip --restart kohadev 9. In one window, run "tcpdump -A -n -v -i any 'port 6001'" 10. In another window, run the following: echo -e "9300CNterm1|COterm1|CPCPL|\r" | nc 127.0.0.1 6001 -v 11. Note in tcpdump output that after the initial flood of packets, ACK packets are sent out every 10+ seconds for the idle connection Signed-off-by: Tadeusz „tadzik” Sośnierz <tadeusz@sosnierz.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37087 HKS3 Tadeusz Sośnierz <tadeusz@sosnierz.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |tadeusz@sosnierz.com Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37087 --- Comment #8 from David Cook <dcook@prosentient.com.au> --- If your SIP server is behind stunnel, you'll need to add at least the following to your stunnel configuration as well: socket = l:SO_KEEPALIVE=1 socket = r:SO_KEEPALIVE=1 If you've tweaked your kernel params at the OS level, then you can leave it there. Otherwise, you'd also need to add stunnel config for TCP_KEEPIDLE and possibly TCP_KEEPINTVL. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37087 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #167713|0 |1 is obsolete| | --- Comment #9 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 169642 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=169642&action=edit Bug 37087: Add TCP keepalive support to SIP server This change adds the ability to enable and configure TCP keepalive support for the SIP server using SIPconfig.xml. For the sake of backwards compatibility, it defaults to disabled and additional parameters default match typical kernel defaults. Technical detail can be found in the perldoc for C4/SIP/SIPserver.pm Test plan: 0. Apply the patch 1. koha-sip --restart kohadev 2. apt-get update && apt-get install tcpdump 3. In one window, run "tcpdump -A -n -v -i any 'port 6001'" 4. In another window, run the following: echo -e "9300CNterm1|COterm1|CPCPL|\r" | nc 127.0.0.1 6001 -v 5. Note in tcpdump output that after the initial flood of packets, nothing more is received 6. vi /etc/koha/sites/kohadev/SIPconfig.xml 7. In the "server-params" element, add attributes like the following: custom_tcp_keepalive='1' custom_tcp_keepalive_time='10' custom_tcp_keepalive_intvl='5' 8. koha-sip --restart kohadev 9. In one window, run "tcpdump -A -n -v -i any 'port 6001'" 10. In another window, run the following: echo -e "9300CNterm1|COterm1|CPCPL|\r" | nc 127.0.0.1 6001 -v 11. Note in tcpdump output that after the initial flood of packets, ACK packets are sent out every 10+ seconds for the idle connection Signed-off-by: Tadeusz „tadzik” Sośnierz <tadeusz@sosnierz.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37087 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |martin.renvoize@ptfs-europe | |.com Status|Signed Off |Passed QA --- Comment #10 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- QA script doesn't like the head level of the POD here.. but there's basically no other POD in the document.. I think this is fine to let through. Code works and is optional. Passing QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37087 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |martin.renvoize@ptfs-europe |y.org |.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37087 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the| |This change adds 3 new release notes| |configuration options to | |the SIPconfig.xml. These | |are custom_tcp_keepalive, | |custom_tcp_keepalive_time, | |and | |custom_tcp_keepalive_intvl. | |Usage is documented in | |C4/SIP/SIPServer.pm. They | |are used to control TCP | |keepalives for the SIP | |server. Configuration of | |these parameters are | |essential for running a SIP | |server in Microsoft Azure. --- Comment #11 from David Cook <dcook@prosentient.com.au> --- RM: let me know if you need anything more for this one. Just added some release note text... -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37087 --- Comment #12 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to David Cook from comment #11)
RM: let me know if you need anything more for this one.
Just added some release note text...
Thanks David. I push in sequence as it's shown on the Dashboard. Getting close to this one. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37087 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the|This change adds 3 new |This change adds 3 new release notes|configuration options to |configuration options to |the SIPconfig.xml. These |the SIPconfig.xml. These |are custom_tcp_keepalive, |are custom_tcp_keepalive, |custom_tcp_keepalive_time, |custom_tcp_keepalive_time, |and |and |custom_tcp_keepalive_intvl. |custom_tcp_keepalive_intvl. |Usage is documented in |Usage is documented in |C4/SIP/SIPServer.pm. They |C4/SIP/SIPServer.pm. They |are used to control TCP |are used to control TCP |keepalives for the SIP |keepalives for the SIP |server. Configuration of |server. Configuration of |these parameters are |these parameters is |essential for running a SIP |essential for running a SIP |server in Microsoft Azure. |server in Microsoft Azure. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37087 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |additional_work_needed --- Comment #13 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Katrin Fischer from comment #12)
(In reply to David Cook from comment #11)
RM: let me know if you need anything more for this one.
Just added some release note text...
Thanks David. I push in sequence as it's shown on the Dashboard. Getting close to this one.
On second thought, would not mind a follow-up making the pod validation happier: FAIL C4/SIP/SIPServer.pm FAIL pod *** WARNING: =head3 without preceding higher level in file C4/SIP/SIPServer.pm *** WARNING: =head3 without preceding higher level in file C4/SIP/SIPServer.pm *** WARNING: =head3 without preceding higher level in file C4/SIP/SIPServer.pm -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37087 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |24.11.00 released in| | Status|Passed QA |Pushed to main -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37087 --- Comment #14 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Pushed for 24.11! Well done everyone, thank you! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37087 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |37696 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37696 [Bug 37696] Improve perldoc for SIP server -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37087 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|additional_work_needed | --- Comment #15 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Thanks for bug 37696, removing additional_work_needed :) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37087 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED Status|Pushed to main |RESOLVED CC| |lucas@bywatersolutions.com --- Comment #16 from Lucas Gass <lucas@bywatersolutions.com> --- Enhancement will not be backported to 24.05.x -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37087 Alex Buckley <alexbuckley@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |alexbuckley@catalyst.net.nz --- Comment #17 from Alex Buckley <alexbuckley@catalyst.net.nz> --- Noting: On the Mattermost town square channel someone has requested that this Bug 37087 (https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37087) be considered for backport to 24.05 in the next round of backports. We have noted to consider its backportability in the January rounds of backports :) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37087 --- Comment #18 from David Cook <dcook@prosentient.com.au> --- (In reply to Alex Buckley from comment #17)
Noting: On the Mattermost town square channel someone has requested that this Bug 37087 (https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37087) be considered for backport to 24.05 in the next round of backports.
We have noted to consider its backportability in the January rounds of backports :)
It should be an easy backport. If anyone is having trouble, happy to lend a hand on this one. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37087 --- Comment #19 from Alex Buckley <alexbuckley@catalyst.net.nz> --- Backported to 24.05.x for 24.05.06 Requested in Mattermost Town Square - see comment #17 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37087 Alex Buckley <alexbuckley@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|24.11.00 |24.11.00, 24.05.06 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37087 --- Comment #20 from Alex Buckley <alexbuckley@catalyst.net.nz> --- There is a request in the Mattermost Town Square for this patchset to be backported to 23.11 - see https://chat.koha-community.org/koha-community/pl/h5k3i77jwfft3q315txh9btxdy -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37087 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|24.11.00, 24.05.06 |24.11.00,24.05.06 released in| | CC| |fridolin.somers@biblibre.co | |m -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37087 --- Comment #21 from Fridolin Somers <fridolin.somers@biblibre.com> --- (In reply to Alex Buckley from comment #20)
There is a request in the Mattermost Town Square for this patchset to be backported to 23.11 - see https://chat.koha-community.org/koha-community/pl/h5k3i77jwfft3q315txh9btxdy
Mmmm, I fell like this can impact the SIP server at deep level. Since selfcheckout machines are a very used feature and 23.11 is now a very stable branch, I prefer not backport to avoid any unseen impact. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37087 --- Comment #22 from David Cook <dcook@prosentient.com.au> --- (In reply to Fridolin Somers from comment #21)
(In reply to Alex Buckley from comment #20)
There is a request in the Mattermost Town Square for this patchset to be backported to 23.11 - see https://chat.koha-community.org/koha-community/pl/h5k3i77jwfft3q315txh9btxdy
Mmmm, I fell like this can impact the SIP server at deep level. Since selfcheckout machines are a very used feature and 23.11 is now a very stable branch, I prefer not backport to avoid any unseen impact.
It can have a significant impact at a low/deep level, but for existing systems (ie upgrades), there's no config change made to SIPconfig.xml, so the functionality isn't activated. Most of the patch is just documentation. If you look at the functions, it's a small amount of code, which I think is clearly explained. I've been running this patch in production with 23.11.x since June 2024 with no problems, and it's essential/critical for running a SIP server in the Azure cloud. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37087 --- Comment #23 from David Cook <dcook@prosentient.com.au> --- (In reply to Fridolin Somers from comment #21)
Mmmm, I fell like this can impact the SIP server at deep level. Since selfcheckout machines are a very used feature and 23.11 is now a very stable branch, I prefer not backport to avoid any unseen impact.
If anything isn't clear about the change, I'm happy to explain it. Overall, it should be a very safe change to make, since it doesn't do anything without adding configuration manually. And even when it's turned on (using custom_tcp_keepalive=1), it will use the default kernel parameters. $ sysctl -a | grep "keepalive" net.ipv4.tcp_keepalive_intvl = 75 net.ipv4.tcp_keepalive_probes = 9 net.ipv4.tcp_keepalive_time = 7200 With these defaults, it won't send a TCP ACK until 2 hours of idle time have passed. If that TCP ACK isn't ACKed back, it will then send up to 9 more ACKs in intervals of 75 seconds. Keep in mind that these are Transport layer packets. They don't contain any actual data in them, so the selfcheckout application and the Koha SIP server application have zero knowledge about them. It's just OS kernel to OS kernel communication. The main use case being to prevent firewalls like Azure's firewall and other corporate firewalls (like Fortinet) from dropping idle persistent SIP TCP connections. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37087 --- Comment #24 from Fridolin Somers <fridolin.somers@biblibre.com> --- OK thanks for the details. I also feel like 23.11.x is too old to have enhancements. Keeping them in upper version encourages people to upgrade, mainly there are several security improvements. And any advanced installation can apply this patch to have the feature. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37087 --- Comment #25 from David Cook <dcook@prosentient.com.au> --- (In reply to Fridolin Somers from comment #24)
OK thanks for the details. I also feel like 23.11.x is too old to have enhancements. Keeping them in upper version encourages people to upgrade, mainly there are several security improvements. And any advanced installation can apply this patch to have the feature.
Hard to argue with that! -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org