25 Nov
2009
25 Nov
'09
6:27 p.m.
On Wed, Nov 25, 2009 at 12:15 PM, Paul Poulain <paul.poulain@biblibre.com>wrote:
Clay Fouts a écrit :
I've discovered that a web crawler was opening up large numbers of simultaneous requests for the RSS/Atom feed. <snip> If anyone can suggest a solution that can keep the system stable while not removing functionality, please let me know.
couldn't a robots.txt do the job ?
Perhaps combined with targeted ipfilter if the crawler disrespects robots.txt.