Some thoughts on the SIP-server scalability
Dear Koha devs! I work at at Oslo public library. We recently decided to go for Koha, and are now working on planning the migration from our current ILS. Our library is fully automated, with RFID-checkin machines taking care of almost all of our circulation. We currently have around 50 of these running at 15 branches around the city. Because of this, we naturally need the SIP-server to handle quite a load. I checked the logs of our current ILS, and found the number of SIP-messages peeked at 58 per second. (In addition to the checkin-machines - we have a lot of RFID-workstations which potentially also will make use of SIP) Correct me if I'm wrong, but this is what I have found out: Koha's SIP server can only handle one request at a time. So if you need more you have to spin up more SIP-severes. This fine for a few, but try to spin up 30 of theese - it takes 15 minutes just to get them all started on a new box with plenty of RAM & CPU. The servers are independent processes which share no resources.. Instead of forking seperate processes for each connection - what about a server which runs as _one_ process and handles multiple tcp connections? I've read about and tested Poe (http://poe.perl.org/) with this in mind - and it seems pretty nice. Poe is an evented framework (akin to Node for javascript or EventMachine in Ruby) which handles such multiplexing of severeal connections. Poe seems very mature, but I'm not competent to judge if its a good fit. There are other options as well (ex IO::Async & AnyEvent ) Any thoughts? We're also very keen on hearing about experiences from other automated Koha libraries, especially those with more than a checkin-machines. Best regards, Petter Goksøyr Åsen Deichmanske bibliotek / Oslo Public Library
On 1 February 2014 01:15, Petter Goksøyr Åsen <petter.goksoyr.asen@kul.oslo.kommune.no> wrote:
Dear Koha devs!
I work at at Oslo public library. We recently decided to go for Koha, and are now working on planning the migration from our current ILS.
Our library is fully automated, with RFID-checkin machines taking care of almost all of our circulation. We currently have around 50 of these running at 15 branches around the city. Because of this, we naturally need the SIP-server to handle quite a load. I checked the logs of our current ILS, and found the number of SIP-messages peeked at 58 per second.
(In addition to the checkin-machines - we have a lot of RFID-workstations which potentially also will make use of SIP)
Correct me if I'm wrong, but this is what I have found out: Koha's SIP server can only handle one request at a time. So if you need more you have to spin up more SIP-severes. This fine for a few, but try to spin up 30 of theese - it takes 15 minutes just to get them all started on a new box with plenty of RAM & CPU. The servers are independent processes which share no resources..
Hmmm, could I see your config please? Because this doesnt sound right at all. What do you have in your SIPconfig.xml I just edited mine to have min 30, spare 1 add it started in seconds Chris
IPConfig below min_spare_servrers was set to 0, we only changed min_servers to 30 I should clarify : I made stress-test with up to 100 concurrent workers sending away checkin & checkout SIP messages. And while the SIP servers start up quickliy, they didnt all accept tcp connections after about 15 minutes.. Anyway, it seems very innefficient to me to have all those independent processes running, with no sharing of resources. By creating a hub with a pool of only 10 sip connections I could easily handle 100 concurrent workers with hundreds of sip transactions per second, so it shouldnt really be a problem w.r.t MySQL to handle that big load. <acsconfig xmlns="http://openncip.org/acs-config/1.0/"> <!-- above address gets nothing, it's just a namespace --> <error-detect enabled="true" /> <!-- Set Net::Server::PreFork runtime parameters syslog_ident will identify SIP2 Koha server entries in syslog For OpenSolaris, add: syslog_logsock=stream --> <server-params min_servers='30' min_spare_servers='0' log_file='Sys::Syslog' syslog_ident='koha_sip' syslog_facility='local6' /> <listeners> <!-- vestigial HTTP, never implemented: just use the OPAC! <service port="0:8080/tcp" transport="http" protocol="NCIP/1.0" /> --> <service port="8023/tcp" transport="telnet" protocol="SIP/2.00" timeout="60" /> <!-- external listener --> <service port="10.172.2.160:6001/tcp" transport="RAW" protocol="SIP/2.00" timeout="60" /> </listeners> <accounts> <login encoding="utf8" id="autohoved1" password="pass" delimiter="|" error-detect="enabled" institution="HUTL" /> <!-- <login id="koha" password="koha" delimiter="|" error-detect="enabled" institution="kohalibrary" /> <login id="koha2" password="koha" institution="kohalibrary2" /> --> <!-- stresstest automats --> <login encoding="utf8" id="stresstest1" password="stresstest1" delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest2" password="stresstest2" delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest3" password="stresstest3" delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest4" password="stresstest4" delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest5" password="stresstest5" delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest6" password="stresstest6" delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest7" password="stresstest7" delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest8" password="stresstest8" delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest9" password="stresstest9" delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest10 " password="stresstest10 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest11 " password="stresstest11 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest12 " password="stresstest12 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest13 " password="stresstest13 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest14 " password="stresstest14 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest15 " password="stresstest15 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest16 " password="stresstest16 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest17 " password="stresstest17 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest18 " password="stresstest18 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest19 " password="stresstest19 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest20 " password="stresstest20 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest21 " password="stresstest21 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest22 " password="stresstest22 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest23 " password="stresstest23 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest24 " password="stresstest24 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest25 " password="stresstest25 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest26 " password="stresstest26 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest27 " password="stresstest27 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest28 " password="stresstest28 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest29 " password="stresstest29 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest30 " password="stresstest30 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest31 " password="stresstest31 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest32 " password="stresstest32 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest33 " password="stresstest33 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest34 " password="stresstest34 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest35 " password="stresstest35 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest36 " password="stresstest36 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest37 " password="stresstest37 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest38 " password="stresstest38 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest39 " password="stresstest39 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest40 " password="stresstest40 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest41 " password="stresstest41 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest42 " password="stresstest42 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest43 " password="stresstest43 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest44 " password="stresstest44 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest45 " password="stresstest45 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest46 " password="stresstest46 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest47 " password="stresstest47 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest48 " password="stresstest48 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest49 " password="stresstest49 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest50 " password="stresstest50 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest51 " password="stresstest51 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest52 " password="stresstest52 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest53 " password="stresstest53 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest54 " password="stresstest54 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest55 " password="stresstest55 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest56 " password="stresstest56 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest57 " password="stresstest57 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest58 " password="stresstest58 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest59 " password="stresstest59 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest60 " password="stresstest60 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest61 " password="stresstest61 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest62 " password="stresstest62 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest63 " password="stresstest63 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest64 " password="stresstest64 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest65 " password="stresstest65 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest66 " password="stresstest66 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest67 " password="stresstest67 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest68 " password="stresstest68 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest69 " password="stresstest69 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest70 " password="stresstest70 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest71 " password="stresstest71 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest72 " password="stresstest72 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest73 " password="stresstest73 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest74 " password="stresstest74 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest75 " password="stresstest75 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest76 " password="stresstest76 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest77 " password="stresstest77 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest78 " password="stresstest78 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest79 " password="stresstest79 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest80 " password="stresstest80 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest81 " password="stresstest81 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest82 " password="stresstest82 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest83 " password="stresstest83 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest84 " password="stresstest84 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest85 " password="stresstest85 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest86 " password="stresstest86 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest87 " password="stresstest87 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest88 " password="stresstest88 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest89 " password="stresstest89 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest90 " password="stresstest90 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest91 " password="stresstest91 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest92 " password="stresstest92 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest93 " password="stresstest93 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest94 " password="stresstest94 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest95 " password="stresstest95 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest96 " password="stresstest96 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest97 " password="stresstest97 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest98 " password="stresstest98 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest99 " password="stresstest99 " delimiter="|" error-detect="enabled" institution="HUTL" /> <login encoding="utf8" id="stresstest100" password="stresstest100" delimiter="|" error-detect="enabled" institution="HUTL" /> </accounts> <!-- Institution tags are for enabled branches. There needs to be one institution stanza for each institution named in the accounts above. The implementation attribute is actually used to find the code to run, in our case "ILS". --> <institutions> <institution id="HUTL" implementation="ILS" parms=""> <policy checkin="true" renewal="true" checkout="true" status_update="false" offline="false" timeout="100" retries="5" /> </institution> <!-- <institution id="CPL" implementation="ILS" parms=""> <policy checkin="true" renewal="true" checkout="true" status_update="false" offline="false" timeout="25" retries="5" /> </institution> --> </institutions> </acsconfig> Petter Goksøyr Åsen Deichmanske bibliotek / Oslo Public Library ________________________________________ Fra: Chris Cormack [chris@bigballofwax.co.nz] Sendt: 31. januar 2014 15:43 Til: Petter Goksøyr Åsen Kopi: koha-devel@lists.koha-community.org Emne: Re: [Koha-devel] Some thoughts on the SIP-server scalability On 1 February 2014 01:15, Petter Goksøyr Åsen <petter.goksoyr.asen@kul.oslo.kommune.no> wrote:
Dear Koha devs!
I work at at Oslo public library. We recently decided to go for Koha, and are now working on planning the migration from our current ILS.
Our library is fully automated, with RFID-checkin machines taking care of almost all of our circulation. We currently have around 50 of these running at 15 branches around the city. Because of this, we naturally need the SIP-server to handle quite a load. I checked the logs of our current ILS, and found the number of SIP-messages peeked at 58 per second.
(In addition to the checkin-machines - we have a lot of RFID-workstations which potentially also will make use of SIP)
Correct me if I'm wrong, but this is what I have found out: Koha's SIP server can only handle one request at a time. So if you need more you have to spin up more SIP-severes. This fine for a few, but try to spin up 30 of theese - it takes 15 minutes just to get them all started on a new box with plenty of RAM & CPU. The servers are independent processes which share no resources..
Hmmm, could I see your config please? Because this doesnt sound right at all. What do you have in your SIPconfig.xml I just edited mine to have min 30, spare 1 add it started in seconds Chris
* Petter Goksøyr Åsen (petter.goksoyr.asen@kul.oslo.kommune.no) wrote:
IPConfig below min_spare_servrers was set to 0, we only changed min_servers to 30
I should clarify : I made stress-test with up to 100 concurrent workers sending away checkin & checkout SIP messages. And while the SIP servers start up quickliy, they didnt all accept tcp connections after about 15 minutes..
Can you humour me, and do min 30, min spare 2 .. and try your stress test again, i'd be interested to see the results. You may still want to rewrite the code to use POE or AnyEvent for other efficiences as you mentioned. Chris -- Chris Cormack Catalyst IT Ltd. +64 4 803 2238 PO Box 11-053, Manners St, Wellington 6142, New Zealand
Can you humour me, and do min 30, min spare 2 .. and try your stress test again, i'd be interested to see the results.
Definetly! But I'm off now, I'l do it when I'm back to work on monday.
You may still want to rewrite the code to use POE or AnyEvent for other efficiences as you mentioned.
Probably not:) Was just a thought - and I probably have no what I'm talking about Just to clarify;Koha easily passed our stress test 4 times the anticipated load :) Have a great weekend! Petter Goksøyr Åsen Deichmanske bibliotek / Oslo Public Library
Can you humour me, and do min 30, min spare 2 .. and try your stress test again, i'd be interested to see the results.
Setting min_spare to 2 did the trick! The SIP-server accept 30 connections within one minute. I'm more relaxed about ths now:) Thanks! Petter Goksøyr Åsen Deichmanske bibliotek / Oslo Public Library
On Fri, Jan 31, 2014 at 01:15:20PM +0100, Petter Goksøyr Åsen wrote:
Correct me if I'm wrong, but this is what I have found out: Koha's SIP server can only handle one request at a time. So if you need more you have to spin up more SIP-severes. This fine for a few, but try to spin up 30 of theese - it takes 15 minutes just to get them all started on a new box with plenty of RAM & CPU.
I'm not seeing these kind of startup times and we have a couple of sites running between 100 and 150 instances. Are you running these up on one port? In practice we tend to assign multiple ports to sip because its easier tracking back and dealing with any networking issues. (i.e. we have a port for branch x and if there are network issues there its easier to track them and restart that port separately). The SIPServer listener forks a separate process to handle a connection which is more efficient than using a POE type architecture. Where that could be improved is allowing some reaping of inactive processes where the client device does not close the connection with a log out. Colin -- Colin Campbell Chief Software Engineer, PTFS Europe Limited Content Management and Library Solutions +44 (0) 800 756 6803 (phone) +44 (0) 7759 633626 (mobile) colin.campbell@ptfs-europe.com skype: colin_campbell2 http://www.ptfs-europe.com
I'm not seeing these kind of startup times and we have a couple of sites running between 100 and 150 instances. Are you running these up on one port? In practice we tend to assign multiple ports to sip because its easier tracking back and dealing with any networking issues. (i.e. we have a port for branch x and if there are network issues there its easier to track them and restart that port separately).
Great! Thats reassuring:) We have only used one port until now, so thanks for the tip! Will try that.
The SIPServer listener forks a separate process to handle a connection which is more efficient than using a POE type architecture. Where that could be improved is allowing some reaping of inactive processes where the client device does not close the connection with a log out.
I trust you on that. Best regards, Petter Goksøyr Åsen Deichmanske bibliotek / Oslo Public Library
participants (4)
-
Chris Cormack -
Chris Cormack -
Colin Campbell -
Petter Goksøyr Åsen