[Koha-devel] git.koha.org outage

[Clay Fouts]

Considering it takes 4.5 minutes for a single instance of the request
to complete, I think a real solution lies in figuring out why
gitweb/git is consuming so many resources in the first place.

Also, I was using the term 'web crawler' loosely in a way that's
inclusive with automated feed readers, the bulk of which don't appear
to request the robots.txt file, and some of which don't even stop
themselves from issuing simultaneous requests for the same feed
(which, I imagine, is because they're not programmed to handle a feed
that takes over four minutes to pull).

I've added the robots.txt because it certainly can't hurt, and I can
analyze the requests after a couple of days. However, I don't think
it's real solution, either.

Clay


On Wed, Nov 25, 2009 at 9:15 AM, Paul Poulain <paul.poulain at biblibre.com> wrote:
> Clay Fouts a écrit :
>>
>> I've discovered that a web crawler was opening up large numbers of
>> simultaneous requests for the RSS/Atom feed.
>
> <snip>
>>
>>  If anyone can suggest a solution that can keep
>> the system stable while not removing functionality, please let me
>> know.
>>
>
> couldn't a robots.txt do the job ?
>
> --
> Paul POULAIN
> http://www.biblibre.com
> Expert en Logiciels Libres pour l'info-doc
> Tel : (33) 4 91 81 35 08
>
>