On Wed, Nov 25, 2009 at 12:09 PM, Clay Fouts <clay.fouts@liblime.com> wrote:
I've discovered that a web crawler was opening up large numbers of simultaneous requests for the RSS/Atom feed. Each of these requests runs an instance of git-diff-tree, which consumes quite a lot of memory and CPU time. Enough of these requests pile up, and the kernel invokes its oom-killer to take care of business. This often times has the unfortunate side effect of killing off key processes, like apache and SSH.
Big ouch!
In order to have another repeat of this, I've disabled gitweb's RSS/Atom feed functionality. It's a little ham-fisted, but not being a git expert, it's the work-around that is most obvious to me right now to stabilize things. If anyone can suggest a solution that can keep the system stable while not removing functionality, please let me know.
Clay
You want a robots.txt file in your root directory to tell robots to not index the git repository. http://en.wikipedia.org/wiki/Robots_exclusion_standard http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449 http://www.robotstxt.org/ Best regards, gvb [snip]