https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42949 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |kyle@bywatersolutions.com |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42949 --- Comment #1 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 201339 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201339&action=edit Bug 42949: Add ability to send a complexity score header for reverse proxy rate limiting Currently, automated scrapers and bots walk the OPAC and hit the most expensive endpoints ( catalog search, record detail ) over and over, which can slow Koha down for everyone. A reverse proxy or WAF such as Cloudflare can rate limit by request count, but it treats every request the same. A flat limit either throttles normal patrons who are browsing, or lets scrapers concentrate on the heavy pages without ever reaching the limit. This patch adds an opt-in response header that weights each request by how expensive the requested endpoint is. Heavy endpoints get a higher score so they consume more of a client's rate limit budget. Cloudflare Rate Limiting rules ( or any WAF that can read a header ) can then throttle the expensive pages harder while leaving the light pages alone. This is modeled on a similar feature in Aspen Discovery. The feature is a Plack middleware ( Koha::Middleware::ComplexityScore ) and is disabled unless it is turned on in koha-conf.xml: <complexity_score_header>1</complexity_score_header> <complexity_score_header_name>x-complexity-score</complexity_score_header_name> complexity_score_header_name is optional and defaults to x-complexity-score. When the feature is off the middleware does nothing. The weights are a static map keyed on the script name, derived from the relative cost of each OPAC script. Sites can override or extend these weights in koha-conf.xml without editing the module. Any script listed in complexity_score_weights replaces its default weight. Scripts that are not listed keep their default: <complexity_score_weights> <opac-search.pl>30</opac-search.pl> <opac-detail.pl>25</opac-detail.pl> </complexity_score_weights> Test Plan: 1) Apply this patch 2) Add to koha-conf.xml, inside the <config> block: <complexity_score_header>1</complexity_score_header> 3) Restart all the things! 4) From a shell, run: curl -sI '<opac>/cgi-bin/koha/opac-search.pl?q=a' | grep -i complexity Note the response has "x-complexity-score: 25"! 5) Repeat for other endpoints and Note the values: * opac-detail.pl?biblionumber=1 => 20 * opac-MARCdetail.pl?biblionumber=1 => 10 * opac-main.pl => 1 6) Add a weights override inside the <config> block: <complexity_score_weights> <opac-search.pl>99</opac-search.pl> </complexity_score_weights> 7) Restart plack again, 8) curl opac-search.pl again 9) Note it now reports 99 while the other endpoints keep their default scores 10) Remove the complexity_score_header line ( or set it to 0 ) 11) Restart plack 12) Curl again 13) Note the x-complexity-score header is gone! -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42949 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42949 --- Comment #3 from David Nind <david@davidnind.com> --- Created attachment 203397 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=203397&action=edit Bug 42949: Add ability to send a complexity score header for reverse proxy rate limiting Currently, automated scrapers and bots walk the OPAC and hit the most expensive endpoints ( catalog search, record detail ) over and over, which can slow Koha down for everyone. A reverse proxy or WAF such as Cloudflare can rate limit by request count, but it treats every request the same. A flat limit either throttles normal patrons who are browsing, or lets scrapers concentrate on the heavy pages without ever reaching the limit. This patch adds an opt-in response header that weights each request by how expensive the requested endpoint is. Heavy endpoints get a higher score so they consume more of a client's rate limit budget. Cloudflare Rate Limiting rules ( or any WAF that can read a header ) can then throttle the expensive pages harder while leaving the light pages alone. This is modeled on a similar feature in Aspen Discovery. The feature is a Plack middleware ( Koha::Middleware::ComplexityScore ) and is disabled unless it is turned on in koha-conf.xml: <complexity_score_header>1</complexity_score_header> <complexity_score_header_name>x-complexity-score</complexity_score_header_name> complexity_score_header_name is optional and defaults to x-complexity-score. When the feature is off the middleware does nothing. The weights are a static map keyed on the script name, derived from the relative cost of each OPAC script. Sites can override or extend these weights in koha-conf.xml without editing the module. Any script listed in complexity_score_weights replaces its default weight. Scripts that are not listed keep their default: <complexity_score_weights> <opac-search.pl>30</opac-search.pl> <opac-detail.pl>25</opac-detail.pl> </complexity_score_weights> Test Plan: 1) Apply this patch 2) Add to koha-conf.xml, inside the <config> block: <complexity_score_header>1</complexity_score_header> 3) Restart all the things! 4) From a shell, run: curl -sI '<opac>/cgi-bin/koha/opac-search.pl?q=a' | grep -i complexity Note the response has "x-complexity-score: 25"! 5) Repeat for other endpoints and Note the values: * opac-detail.pl?biblionumber=1 => 20 * opac-MARCdetail.pl?biblionumber=1 => 10 * opac-main.pl => 1 6) Add a weights override inside the <config> block: <complexity_score_weights> <opac-search.pl>99</opac-search.pl> </complexity_score_weights> 7) Restart plack again, 8) curl opac-search.pl again 9) Note it now reports 99 while the other endpoints keep their default scores 10) Remove the complexity_score_header line ( or set it to 0 ) 11) Restart plack 12) Curl again 13) Note the x-complexity-score header is gone! Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42949 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david@davidnind.com --- Comment #5 from David Nind <david@davidnind.com> --- Testing notes (using KTD): 1. The tests pass: prove t/Koha/Middleware/ComplexityScore.t 2. I put the <complexity_score_header>1</complexity_score_header> before the Elasticsearch configuration, around line 354. -- You are receiving this mail because: You are watching all bug changes.