[Bug 4042] New: Public OPAC search can fall prey to web crawlers
http://bugs.koha.org/cgi-bin/bugzilla3/show_bug.cgi?id=4042 Summary: Public OPAC search can fall prey to web crawlers Product: Koha Version: unspecified Platform: All OS/Version: All Status: NEW Severity: enhancement Priority: P5 Component: OPAC AssignedTo: jmf@liblime.com ReportedBy: rick@praxis.com.au Estimated Hours: 0.0 Change sponsored?: --- The OPAC search and OPAC advanced searches are accessible by the public from the Koha OPAC home page. Consequently, an over zealous web crawler indexing the site using the opac-search.pl script can impact the performance of the Koha system. In the extreme, an under- resourced system can experience a DoS when the number of searches exceeds the capacity of the system. Proposed Solution: modify the opac-search.pl script in the following manner: (A) Only allow queries using the POST method; otherwise if GET is used return a simple page with "No search result found". (B) Exception: do allow GET queries but only if the HTTP_REFERER matches the SERVER_NAME. This allows all the searches to work via web site links. (C) Make this behavior optional by adding a new flag to the system prefs. Here is the small code segment added to opac-search.pl, immediately after the BEGIN block: if ($ENV{HTTP_REFERER} !~ /$ENV{SERVER_NAME}/ && $ENV{REQUEST_METHOD} ne "POST") { print "Content-type: text/html\n\n"; print "<h1>Search Results</h1>Nothing found.\n"; exit; } CAVEAT: This solution does not allow one to paste an "opac_search.pl" link into the browser and have it work as previously expected. But this was the cause of the problem in the first place. A better solution is to require a user to login to the OPAC before allowing a search. Addendum: also install a robots.txt file at the following location in the Koha source tree to stop web crawlers from using the OPAC search. opac/htdocs/robots.txt The robots.txt file should contain the following contents, which deny all access to indexing engines. You can learn more about robots.txt on the web, and configure it to allow some indexing if you wish. ----------------------------- User-agent: * Disallow: / ----------------------------- -- Configure bugmail: http://bugs.koha.org/cgi-bin/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha.org/cgi-bin/bugzilla3/show_bug.cgi?id=4042 --- Comment #1 from Rick Welykochy <rick@praxis.com.au> 2010-01-13 23:25:06 --- The proposed solution has been implemented and tested by Calyx Information Essentials in Australia. We are no longer experiencing any problems relating to web crawlers DoS-ing our Koha server. -- Configure bugmail: http://bugs.koha.org/cgi-bin/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha.org/cgi-bin/bugzilla3/show_bug.cgi?id=4042 --- Comment #2 from Owen Leonard <oleonard@myacpl.org> 2010-01-13 23:44:36 --- Why is the use of robots.txt not enough in and of itself to solve the problem? -- Configure bugmail: http://bugs.koha.org/cgi-bin/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha.org/cgi-bin/bugzilla3/show_bug.cgi?id=4042 Irma Birchall <irma@calyx.net.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |irma@calyx.net.au AssignedTo|jmf@liblime.com |bob@calyx.net.au -- Configure bugmail: http://bugs.koha.org/cgi-bin/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha.org/cgi-bin/bugzilla3/show_bug.cgi?id=4042 --- Comment #3 from Rick Welykochy <rick@praxis.com.au> 2010-01-14 01:02:45 --- 1. robots.txt only influences law-abiding crawlers. 2. Allowing GET searches from outside means that people could save search URLs on their pages or bookmarks. When a private crawler or link checker sees this search URL, it might do a fetch without checking robots.txt. In general public GET URLs that do a significant amount of work are problematic. -- Configure bugmail: http://bugs.koha.org/cgi-bin/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha.org/cgi-bin/bugzilla3/show_bug.cgi?id=4042 --- Comment #4 from Ken Yap <ken@calyx.net.au> 2010-01-15 01:33:17 --- Actually my preference for handing an illegal GET request from outside is to send a redirect to the same URL minus the query parameters. That way, if a human happens to have saved a search result by copying the location bar, they don't get "nothing found" when they use the link but get prompted to enter another query. More useful than a "nothing found" page. Also your "nothing found" message is English-centric and would need translation for sites in other countries. By sending a redirect you don't have to compose any text. http://bugs.koha.org/cgi-bin/bugzilla3/show_bug.cgi?id=4042 -- Configure bugmail: http://bugs.koha.org/cgi-bin/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha.org/cgi-bin/bugzilla3/show_bug.cgi?id=4042 --- Comment #5 from Rick Welykochy <rick@praxis.com.au> 2010-01-15 09:16:09 --- Ken has a good idea. Allow the GET request to be processed, but show the original page, not the search results. To make things easier for the genuine Koha user, i.e. one who will then actually click on the search button (POST) rather than try a GET, populate the form with the query parameters form the GET request it finds in the URL. This saves the user from entering them again. Painless and simple. Overall this is looking like a good strategy to keep misbehaved bots at bay. All this is best done in a subroutine residing somewhere in the C4 lib subdirectory so that it can be used anywhere necessary in Koha. Some admins configure Koha to allow public access to other parts of the OPAC as well. -- Configure bugmail: http://bugs.koha.org/cgi-bin/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha.org/cgi-bin/bugzilla3/show_bug.cgi?id=4042 Nicole C. Engard <nengard@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nengard@gmail.com --- Comment #6 from Nicole C. Engard <nengard@gmail.com> 2010-01-15 12:49:53 --- Correct me if I'm wrong - but this would remove the ability for librarians to send their patrons links to queries that they think will help with their research. This is a huge deal to librarians I have trained. They love this feature, because it's something they have never had before. When I do training I always make it clear that every link in Koha can be copied and pasted into an email and will work exactly as you would expect. So -- I want to bring us all back to a suggestion made on the mailing list or IRC - wherever this discussion started to have a system preference to let the library decide what they want to do. Many libraries want their catalog indexed by all search engines and they want to be able to send links to results lists to their patrons. This is a feature they have now and would miss if you were to take that away. It should be the decision of the librarians and the system admins - not the developers. -- Configure bugmail: http://bugs.koha.org/cgi-bin/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha.org/cgi-bin/bugzilla3/show_bug.cgi?id=4042 Magnus Enger <magnus@enger.priv.no> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |magnus@enger.priv.no --- Comment #7 from Magnus Enger <magnus@enger.priv.no> 2010-01-15 13:04:58 --- I agree completely with Nicole: Being able to send URLs to searches in email or putting them on a webpage is a big feature, not a bug. I also agree that a syspref to turn this behaviour on and off is the way to go. -- Configure bugmail: http://bugs.koha.org/cgi-bin/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha.org/cgi-bin/bugzilla3/show_bug.cgi?id=4042 --- Comment #8 from Rick Welykochy <rick@praxis.com.au> 2010-01-15 13:07:41 ---
Correct me if I'm wrong - but this would remove the ability for librarians to send their patrons links to queries that they think will help with their research.
IRC - wherever this discussion started to have a system preference to let the
Not at all. It has been proposed that there is a system pref to disable this behavior. If the behaviour is enabled, all theuse has to do is hit the "Search" button when they go to a search link. library decide what they want to do. Of course. This is necessary.
Being able to send URLs to searches in email or putting them on a webpage is a big feature, not a bug.
Of course, this is a "bug" repository. But it is much more as well. This fix proproses to *optionally* make new behaviour available in Koha to prevent web crawlers from DOS-ing your system. Never fear. This fix will be optional via System Prefs :) -- Configure bugmail: http://bugs.koha.org/cgi-bin/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha.org/cgi-bin/bugzilla3/show_bug.cgi?id=4042 --- Comment #9 from Nicole C. Engard <nengard@gmail.com> 2010-01-15 13:29:11 --- Awesome! Then I'm all for it ;) -- Configure bugmail: http://bugs.koha.org/cgi-bin/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠kohaorg.ec2.liblime.com