https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=4042 Barry Cannon <bc@interleaf.ie> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bc@interleaf.ie --- Comment #12 from Barry Cannon <bc@interleaf.ie> --- Reading over this bug it occurs to me that some of the steps we have taken to mitigate this problem might help others. I will post high level information here, in case it might help some people along. The first step we took (after adding robots.txt) was to add a small piece of code to the OPACHeader syspref that appended a “hidden” a href tag to a script (sneakysnare.pl) on the site. This would be only visible to bots and once this link was followed a page would show with a lot of useless text. In the background though the script grabbed the source IP address and pushed it into a deny rule on the firewall. This worked well for bots that blindly followed all links from the page. Shortly after we noticed that not all bots were following all links. Our next step was to check the useragent of the incoming traffic. We noticed that there were a lot of useragent strings causing issues. Configuring apache for a CustomLog of “time,IP,METHOD,URI” we configured a script to run regularly to parse this file for know “bad” useragent strings. We were then able to add these IPs to the firewall to be dropped. Our current setup is expanded from above: all our servers use csf/lfd for local firewall and intrusion detection. We also use ipset extensively for other non-Koha related services. Csf can be configured to offload deny chains to ipset. This helps iptables and lowers the resource strain on the server. Csf can be configured to use an Include file to deny hosts. By expanding on the sneakysnare script and the useragent apache log we create a small job to bring all this together and manage an csf include file. The job checks this new file and if a new IP address has appeared it will add that IP to the deny set. In some cases we have observed the server being slowly scrapped. This insidious scrapping is harder to detect immediately and often slows/hogs resources over a longer period. Quite often the source of these connections are from a particular geographical region. If this happens - often enough - we can employ geoblocking. Csf can be configured to use Maxmind GeoIP database lookup. Using the configuration file we specify the country code we want to block. For example to block all Irish and British traffic (not that we would!) we enter the “IE,GB” into the config file. Once the daemon is restarted the GeoIP database is referenced and all known CIDR blocks for those nountries are loaded into ipset’s deny set. Csf can also be configured to “deny all, except”. It this setup placing “IE” in the config file would only allow traffic from Ireland and deny all other traffic. There are pros and cons to all of the above and consideration should be given before implementation. Third party services are also very useful and moving traffic via CDN providers (and using their security services) will greatly reduce bots, DDOS and other hassle. Other helpful methods include reverse proxies and mitigating at that level. -- You are receiving this mail because: You are watching all bug changes. You are the QA Contact for the bug.