Hi, all. Just a brief update on the git.koha.org outage. That server is currently unreachable, even on an administrative basis. We've been working with the hosting provider (Spry) for the past several hours to get it back online. It ought to be available again soon. Cheers, Clay
2009/11/25 Clay Fouts <clay.fouts@liblime.com>:
Hi, all.
Just a brief update on the git.koha.org outage. That server is currently unreachable, even on an administrative basis. We've been working with the hosting provider (Spry) for the past several hours to get it back online. It ought to be available again soon.
Hi Clay Thanks very much for the report. Outages like these are just one more reason that I love git, back when we used cvs, this would bring things to a grinding halt, but the fact that everyone has their own repository with git, it is nowhere near as bad Chris
Hi Clay, On Tue, Nov 24, 2009 at 3:53 PM, Clay Fouts <clay.fouts@liblime.com> wrote:
Hi, all.
Just a brief update on the git.koha.org outage. That server is currently unreachable, even on an administrative basis. We've been working with the hosting provider (Spry) for the past several hours to get it back online. It ought to be available again soon.
Thanks for the help with this! Kind Regards, Chris
Git service is now fully restored. I'm doing a post-mortem and will report back on the cause if there's anything unusual. Clay On Tue, Nov 24, 2009 at 12:53 PM, Clay Fouts <clay.fouts@liblime.com> wrote:
Hi, all.
Just a brief update on the git.koha.org outage. That server is currently unreachable, even on an administrative basis. We've been working with the hosting provider (Spry) for the past several hours to get it back online. It ought to be available again soon.
Cheers, Clay
I was able to get to it last night - but it's not working again this AM :( Nicole On Tue, Nov 24, 2009 at 5:10 PM, Clay Fouts <clay.fouts@liblime.com> wrote:
Git service is now fully restored. I'm doing a post-mortem and will report back on the cause if there's anything unusual.
Clay
On Tue, Nov 24, 2009 at 12:53 PM, Clay Fouts <clay.fouts@liblime.com> wrote:
Hi, all.
Just a brief update on the git.koha.org outage. That server is currently unreachable, even on an administrative basis. We've been working with the hosting provider (Spry) for the past several hours to get it back online. It ought to be available again soon.
Cheers, Clay
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
I've discovered that a web crawler was opening up large numbers of simultaneous requests for the RSS/Atom feed. Each of these requests runs an instance of git-diff-tree, which consumes quite a lot of memory and CPU time. Enough of these requests pile up, and the kernel invokes its oom-killer to take care of business. This often times has the unfortunate side effect of killing off key processes, like apache and SSH. In order to have another repeat of this, I've disabled gitweb's RSS/Atom feed functionality. It's a little ham-fisted, but not being a git expert, it's the work-around that is most obvious to me right now to stabilize things. If anyone can suggest a solution that can keep the system stable while not removing functionality, please let me know. Clay On Wed, Nov 25, 2009 at 5:43 AM, Nicole Engard <nengard@gmail.com> wrote:
I was able to get to it last night - but it's not working again this AM :(
Nicole
On Tue, Nov 24, 2009 at 5:10 PM, Clay Fouts <clay.fouts@liblime.com> wrote:
Git service is now fully restored. I'm doing a post-mortem and will report back on the cause if there's anything unusual.
Clay
On Tue, Nov 24, 2009 at 12:53 PM, Clay Fouts <clay.fouts@liblime.com> wrote:
Hi, all.
Just a brief update on the git.koha.org outage. That server is currently unreachable, even on an administrative basis. We've been working with the hosting provider (Spry) for the past several hours to get it back online. It ought to be available again soon.
Cheers, Clay
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
Clay Fouts a écrit :
I've discovered that a web crawler was opening up large numbers of simultaneous requests for the RSS/Atom feed. <snip> If anyone can suggest a solution that can keep the system stable while not removing functionality, please let me know.
couldn't a robots.txt do the job ? -- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc Tel : (33) 4 91 81 35 08
On Wed, Nov 25, 2009 at 12:15 PM, Paul Poulain <paul.poulain@biblibre.com>wrote:
Clay Fouts a écrit :
I've discovered that a web crawler was opening up large numbers of simultaneous requests for the RSS/Atom feed. <snip> If anyone can suggest a solution that can keep the system stable while not removing functionality, please let me know.
couldn't a robots.txt do the job ?
Perhaps combined with targeted ipfilter if the crawler disrespects robots.txt.
Considering it takes 4.5 minutes for a single instance of the request to complete, I think a real solution lies in figuring out why gitweb/git is consuming so many resources in the first place. Also, I was using the term 'web crawler' loosely in a way that's inclusive with automated feed readers, the bulk of which don't appear to request the robots.txt file, and some of which don't even stop themselves from issuing simultaneous requests for the same feed (which, I imagine, is because they're not programmed to handle a feed that takes over four minutes to pull). I've added the robots.txt because it certainly can't hurt, and I can analyze the requests after a couple of days. However, I don't think it's real solution, either. Clay On Wed, Nov 25, 2009 at 9:15 AM, Paul Poulain <paul.poulain@biblibre.com> wrote:
Clay Fouts a écrit :
I've discovered that a web crawler was opening up large numbers of simultaneous requests for the RSS/Atom feed.
<snip>
If anyone can suggest a solution that can keep the system stable while not removing functionality, please let me know.
couldn't a robots.txt do the job ?
-- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc Tel : (33) 4 91 81 35 08
Hi Clay, On Wed, Nov 25, 2009 at 12:09 PM, Clay Fouts <clay.fouts@liblime.com> wrote:
I've discovered that a web crawler was opening up large numbers of simultaneous requests for the RSS/Atom feed. Each of these requests runs an instance of git-diff-tree, which consumes quite a lot of memory and CPU time. Enough of these requests pile up, and the kernel invokes its oom-killer to take care of business. This often times has the unfortunate side effect of killing off key processes, like apache and SSH.
In order to have another repeat of this, I've disabled gitweb's RSS/Atom feed functionality. It's a little ham-fisted, but not being a git expert, it's the work-around that is most obvious to me right now to stabilize things. If anyone can suggest a solution that can keep the system stable while not removing functionality, please let me know.
Thanks for the time invested in investigating and fixing this issue! Kind Regards, Chris
On Wed, Nov 25, 2009 at 12:09 PM, Clay Fouts <clay.fouts@liblime.com> wrote:
I've discovered that a web crawler was opening up large numbers of simultaneous requests for the RSS/Atom feed. Each of these requests runs an instance of git-diff-tree, which consumes quite a lot of memory and CPU time. Enough of these requests pile up, and the kernel invokes its oom-killer to take care of business. This often times has the unfortunate side effect of killing off key processes, like apache and SSH.
Big ouch!
In order to have another repeat of this, I've disabled gitweb's RSS/Atom feed functionality. It's a little ham-fisted, but not being a git expert, it's the work-around that is most obvious to me right now to stabilize things. If anyone can suggest a solution that can keep the system stable while not removing functionality, please let me know.
Clay
You want a robots.txt file in your root directory to tell robots to not index the git repository. http://en.wikipedia.org/wiki/Robots_exclusion_standard http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449 http://www.robotstxt.org/ Best regards, gvb [snip]
Did we ever come up with another solution to this?? I used the RSS feed to keep track of things - for me, I just prefer it over the email list. Nicole On Wed, Nov 25, 2009 at 3:34 PM, Gerald Van Baren <vanbargw@gmail.com> wrote:
On Wed, Nov 25, 2009 at 12:09 PM, Clay Fouts <clay.fouts@liblime.com> wrote:
I've discovered that a web crawler was opening up large numbers of simultaneous requests for the RSS/Atom feed. Each of these requests runs an instance of git-diff-tree, which consumes quite a lot of memory and CPU time. Enough of these requests pile up, and the kernel invokes its oom-killer to take care of business. This often times has the unfortunate side effect of killing off key processes, like apache and SSH.
Big ouch!
In order to have another repeat of this, I've disabled gitweb's RSS/Atom feed functionality. It's a little ham-fisted, but not being a git expert, it's the work-around that is most obvious to me right now to stabilize things. If anyone can suggest a solution that can keep the system stable while not removing functionality, please let me know.
Clay
You want a robots.txt file in your root directory to tell robots to not index the git repository. http://en.wikipedia.org/wiki/Robots_exclusion_standard http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449 http://www.robotstxt.org/
Best regards, gvb
[snip]
participants (7)
-
Chris Cormack -
Chris Nighswonger -
Clay Fouts -
Gerald Van Baren -
Joe Atzberger -
Nicole Engard -
Paul Poulain