[Bug 41767] New: Plack backlog parameter
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41767 Bug ID: 41767 Summary: Plack backlog parameter Initiative type: --- Sponsorship --- status: Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Architecture, internals, and plumbing Assignee: koha-bugs@lists.koha-community.org Reporter: andreas.jonsson@kreablo.se QA Contact: testopia@bugs.koha-community.org The default request backlog for starman is 1024 request wich seems too large for Koha. When the system is overloaded it will typically take several minutes to recover because of the backlog of mostly stale requests. I suggest we set a much lower default value and expose a configuration parameter. I propose $PLACK_WORKERS * $PLACK_MAX_REQUESTS / 2 as a default value (just guessing). -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41767 Andreas Jonsson <andreas.jonsson@kreablo.se> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |andreas.jonsson@kreablo.se |ity.org | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41767 Andreas Jonsson <andreas.jonsson@kreablo.se> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff Patch complexity|--- |Trivial patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41767 --- Comment #1 from Andreas Jonsson <andreas.jonsson@kreablo.se> --- Created attachment 192449 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=192449&action=edit Bug 41767: Plack backlog parameter Set a low default --backlog parameter for starman to allow for faster recovery when system becomes overloaded. Also expose configuration parameter for tuning. Test plan: * At the command line verify that the parameter "--backlog 50" is present for starman processes, provided that --max-requests 50 --workers 2: > ps aux | grep starman * Edit /etc/koha/sites/kohadev/koha-conf.xml and add <plack_backlog>157</plack_backlog> (e.g., next to <plack_workers>). * Restart plack: > koha-plack --restart kohadev * Verify that the --backlog parameter has changed accordingly -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41767 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41767
David Nind <david@davidnind.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #192449|0 |1
is obsolete| |
--- Comment #2 from David Nind <david@davidnind.com> ---
Created attachment 194208
-->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=194208&action=edit
Bug 41767: Plack backlog parameter
Set a low default --backlog parameter for starman to allow for faster
recovery when system becomes overloaded. Also expose configuration
parameter for tuning.
Test plan:
* At the command line verify that the parameter "--backlog 50" is
present for starman processes, provided that --max-requests 50
--workers 2:
> ps aux | grep starman
* Edit /etc/koha/sites/kohadev/koha-conf.xml and add
<plack_backlog>157</plack_backlog> (e.g., next to <plack_workers>).
* Restart plack:
> koha-plack --restart kohadev
* Verify that the --backlog parameter has changed accordingly
Signed-off-by: David Nind <david@davidnind.com>
--
You are receiving this mail because:
You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41767 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david@davidnind.com Text to go in the| |The default request backlog release notes| |for starman (Plack (?)) is | |1024 requests, which seems | |too large for Koha. When | |the system is overloaded it | |will typically take several | |minutes to recover because | |of the backlog of mostly | |stale requests. | | | |This enhancement: | |- sets the default value to | |$PLACK_WORKERS * | |$PLACK_MAX_REQUESTS / 2 | |- lets libraries set their | |own default --backlog | |parameter for starman, to | |allow for faster recovery | |when systems become | |overloaded. | | | |Existing installations will | |need to add an entry to | |their instance's | |koha-conf.xml file to | |change the default, for | |example: | |<plack_backlog>50</plack_ba | |cklog> | | | |For new installations, | |commented out settings for | |plack to limit the backlog | |is available in their | |instance's koha-conf.xml: | | | |<!-- The parameter | |plack_backlog is the size | |of the request backlog | |queue. --> | |<!-- The default is | |(max_requests*workers)/2 | |--> | |<!-- | |<plack_backlog>50</plack_ba | |cklog> --> --- Comment #3 from David Nind <david@davidnind.com> --- I've attempted a release note, but not sure that I got it right. Note that if I start up KTD with the patch applied, the KTD /etc/koha/sites/kohadev/koha-conf.xml does not have the commented out settings. Should this be the case or not? Or would this need to be a change made to the template for koha-testing-docker. Testing notes (using KTD): 1. Apply the patch. 2. Reset everything: reset_all 3. Add the plack entry to koha-conf.xml as per the test plan (Search for <plack to add near the existing plack settings). -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41767 Andreas Jonsson <andreas.jonsson@kreablo.se> changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the|The default request backlog |The default request backlog release notes|for starman (Plack (?)) is |for starman (the Plack |1024 requests, which seems |server) is 1024 requests, |too large for Koha. When |which seems too large for |the system is overloaded it |Koha. When the system is |will typically take several |overloaded it will |minutes to recover because |typically take several |of the backlog of mostly |minutes to recover because |stale requests. |of the backlog of mostly | |stale requests. |This enhancement: | |- sets the default value to |This enhancement: |$PLACK_WORKERS * |- sets the default value to |$PLACK_MAX_REQUESTS / 2 |$PLACK_WORKERS * |- lets libraries set their |$PLACK_MAX_REQUESTS / 2 |own default --backlog |- lets libraries set their |parameter for starman, to |own --backlog parameter for |allow for faster recovery |starman via koha-conf.xml. |when systems become | |overloaded. |Existing installations will | |need to add an entry to |Existing installations will |their instance's |need to add an entry to |koha-conf.xml file if there |their instance's |is need to change the |koha-conf.xml file to |setting from the default, |change the default, for |for example: |example: |<plack_backlog>50</plack_ba |<plack_backlog>50</plack_ba |cklog> |cklog> | | |For new installations, |For new installations, |commented out settings for |commented out settings for |plack to limit the backlog |plack to limit the backlog |is available in their |is available in their |instance's koha-conf.xml: |instance's koha-conf.xml: | | |<!-- The parameter |<!-- The parameter |plack_backlog is the size |plack_backlog is the size |of the request backlog |of the request backlog |queue. --> |queue. --> |<!-- The default is |<!-- The default is |(max_requests*workers)/2 |(max_requests*workers)/2 |--> |--> |<!-- |<!-- |<plack_backlog>50</plack_ba |<plack_backlog>50</plack_ba |cklog> --> |cklog> --> | --- Comment #4 from Andreas Jonsson <andreas.jonsson@kreablo.se> --- Thanks you David! I reworded the release note text slightly to make it more clear that you don't need to do anything unless there's a specific need. I don't know how koha-conf.xml is managed in the docker images, but I would guess it is regenerated from the template when the image is rebuilt. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41767 Thomas Klausner <domm@plix.at> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |domm@plix.at |y.org | CC| |domm@plix.at -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41767 Thomas Klausner <domm@plix.at> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA --- Comment #5 from Thomas Klausner <domm@plix.at> --- To test this in KTD, I had to `sudo cp debian/scripts/koha-plack /usr/sbin/koha-plack`, but after that it worked like a charm. I did not test the actual effect of setting --backlog, but trust Andreas :-) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41767 Thomas Klausner <domm@plix.at> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #194208|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41767 --- Comment #6 from Thomas Klausner <domm@plix.at> --- Created attachment 196443 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=196443&action=edit Bug 41767: Plack backlog parameter Set a low default --backlog parameter for starman to allow for faster recovery when system becomes overloaded. Also expose configuration parameter for tuning. Test plan: * At the command line verify that the parameter "--backlog 50" is present for starman processes, provided that --max-requests 50 --workers 2: > ps aux | grep starman * Edit /etc/koha/sites/kohadev/koha-conf.xml and add <plack_backlog>157</plack_backlog> (e.g., next to <plack_workers>). * Restart plack: > koha-plack --restart kohadev * Verify that the --backlog parameter has changed accordingly Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Thomas Klausner <domm@plix.at> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41767 Lucas Gass (lukeg) <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lucas@bywatersolutions.com Status|Passed QA |Failed QA --- Comment #7 from Lucas Gass (lukeg) <lucas@bywatersolutions.com> --- This needs more consideration. We cannot "just guess" what a good default lower value is. We need to see some testing and research to prove what is best. I also trust Andreas but that isn't how QA works. This must be fully tested with some metrics before patch vs after patch. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41767 --- Comment #8 from Lucas Gass (lukeg) <lucas@bywatersolutions.com> --- I think this needs more thought: $PLACK_WORKERS * $PLACK_MAX_REQUESTS / 2 Example 1: plack_requests: 50 plack_workers: 6 300 / 2 = 150 which seems too low Example 2: plack_requests: 200 plack_workers: 32 32 * 200 = 6400 / 2 = 3200 which seems really high. Much higher than the default. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41767 --- Comment #9 from Andreas Jonsson <andreas.jonsson@kreablo.se> --- I think that the default value should only be dependent of the number of Plack workers. Maybe it should be on the form C1 + C2 * $PLACK_WORKERS where C1 and C2 are some well considered constants. For instance, C1 = 50, and C2 = 20. I think that a lower value is better but it is safer to start high until we have some more data. I have verified that the backlog-parameter sets the Listen parameter on IO::Socket::UNIX. Since the socket is opened in the master process the queue should be shared by all workers. We also need to consider how to configure Apache for this to be really effective as Apache also seems to be queueing requests. Maybe a lower ProxyTimeout is needed. -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org