[Koha-devel] git.koha.org outage

[Gerald Van Baren]

On Wed, Nov 25, 2009 at 12:09 PM, Clay Fouts <clay.fouts at liblime.com> wrote:
> I've discovered that a web crawler was opening up large numbers of
> simultaneous requests for the RSS/Atom feed. Each of these requests
> runs an instance of git-diff-tree, which consumes quite a lot of
> memory and CPU time. Enough of these requests pile up, and the kernel
> invokes its oom-killer to take care of business. This often times has
> the unfortunate side effect of killing off key processes, like apache
> and SSH.

Big ouch!

> In order to have another repeat of this, I've disabled gitweb's
> RSS/Atom feed functionality. It's a little ham-fisted, but not being a
> git expert, it's the work-around that is most obvious to me right now
> to stabilize things. If anyone can suggest a solution that can keep
> the system stable while not removing functionality, please let me
> know.
>
> Clay

You want a robots.txt file in your root directory to tell robots to
not index the git repository.
http://en.wikipedia.org/wiki/Robots_exclusion_standard
http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449
http://www.robotstxt.org/

Best regards,
gvb

[snip]