[Bug 30614] New: check-url-quick gives false error 404
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30614 Bug ID: 30614 Summary: check-url-quick gives false error 404 Change sponsored?: --- Product: Koha Version: 20.05 Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Architecture, internals, and plumbing Assignee: koha-bugs@lists.koha-community.org Reporter: marjorie.barry-vila@collecto.ca QA Contact: testopia@bugs.koha-community.org Hi, Many 404 errors come out after running the check-quick-url script while the urls are correct when checked manually. Apparently this is because the script uses HTTP HEAD instead of HTTP GET. HTTP HEAD returns only the headers of the urls and the header can return an error but the url is accessible when you do a GET. Is there a reason for this? My fellow analyst was pointing to line 104 of https://github.com/Koha-Community/Koha/blob/master/misc/cronjobs/check-url-q... Thank you Marjorie -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30614 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |aleisha@catalyst.net.nz --- Comment #1 from Aleisha Amohia <aleisha@catalyst.net.nz> --- This script is very old, I imagine the way it's being done is a reflection of it's age. We use HTTP::Request throughout Koha now. I imagine replace that with something close to how we do this in Koha/ERM/EUsage/UsageDataProvider.pm : =head3 _handle_sushi_response Creates and sends the request based on a provided url Also handles any redirects =cut Koha/ERM/EUsage/UsageDataProvider.pm sub _handle_sushi_request { my ($url) = @_; my $request = HTTP::Request->new( 'GET' => $url ); my $ua = LWP::UserAgent->new; $ua->agent( 'Koha/' . Koha::version() ); my $response = $ua->simple_request($request); if ( $response->is_redirect ) { my $redirect_url = $response->header('Location'); $redirect_url = URI->new_abs( $redirect_url, $url ); $response = $ua->get($redirect_url); } return $response; } -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30614 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au --- Comment #2 from David Cook <dcook@prosentient.com.au> --- When it comes to checking URLs, typically you want to do a HTTP HEAD instead of a HEAD GET to increase performance for the checker and to decrease load on the URL being checked. For instance, let's say you're checking if URL "https://path/to/1TB/file.file" exists. If you do a HTTP HEAD, that can be a <1 second check. If you do a HTTP GET, you're going to have to wait for that file to download, so your checker will go much slower. It's also going to cost the server host money in terms of data transfer. If you're using cloud like Azure or AWS, your costs can increase by hundreds or thousands of dollars very easily. So HTTP HEAD is much better than HTTP GET for this use case. However... not all sites honour HTTP HEAD. Misguided people trying to lockdown servers for security reasons will sometimes limit HTTP verbs to just GET/POST, but this actually has a negative effect, because it means you then have to do a full HTTP GET to do something like a URL check. I work on a project with millions of URLs and many terabytes of data, and HTTP HEAD is one of my best friends. Anyway... I'd say an improvement to the script would be to add a CLI parameter to choose whether to use a "HEAD" or a "GET" request, because realistically sometimes you do have to use a GET to get an accurate response - even if HEAD is more optimal in an ideal world. Another optimisation could be to fall back to a GET in the event that a HEAD fails. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30614 --- Comment #3 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Fair enough, I like the fall back option. A lot of these third party providers have redirects or bot challenges now that seem to cause HTTP HEAD to fail for some reason -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30614 --- Comment #4 from David Cook <dcook@prosentient.com.au> --- (In reply to Aleisha Amohia from comment #3)
Fair enough, I like the fall back option. A lot of these third party providers have redirects or bot challenges now that seem to cause HTTP HEAD to fail for some reason
Ah right. Of course. Since so many bots start off with HTTP HEAD, it is a natural target for sure. I've run into that with some of my own non-Koha link checkers as well. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30614 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED Assignee|koha-bugs@lists.koha-commun |aleisha@catalyst.net.nz |ity.org | Change sponsored?|--- |Sponsored -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30614 --- Comment #5 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 186380 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=186380&action=edit Sample MARC with error URLs check-url-quick.pl returns false errors for these URLs 439 http://www.tandfonline.com/loi/tejp20 403 Forbidden 441 https://ir.canterbury.ac.nz/handle/10092/1507 403 Forbidden 440 http://onlinelibrary.wiley.com/journal/10.1002/%28ISSN%291552-8618/issues 403 Forbidden 442 https://natlib-primo.hosted.exlibrisgroup.com/permalink/f/1s57t7d/NLNZ_ALMA1... 403 Cache-Control: private 443 https://coldregions.americangeosciences.org/vufind/Content/ajus 404 Not Found -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30614 --- Comment #6 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 186381 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=186381&action=edit Bug 30614: [WIP] Fallback to a GET request if HEAD returns error status This is a WIP - feedback welcome. Sample MARC attached. Testing using: perl misc/cronjobs/check-url-quick.pl -v > test.txt grep -E "tand|wiley|natlib|cold|canterbury" test.txt -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30614 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30614 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #186381|0 |1 is obsolete| | --- Comment #7 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 187030 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=187030&action=edit Bug 30614: Fallback to a GET request if HEAD returns error status This enhancement makes a GET request as a fallback if HEAD returns an error status. This doesn't fix all of the errors produced by the sample error MARC attached - any further support is welcomed. To test: 1. Download MARC. Go to Cataloguing in the staff interface and stage the MARC file for import, then import. 2. Run the URL check job and confirm erroring URLs perl misc/cronjobs/check-url-quick.pl -v > test.txt grep -E "tand|wiley|natlib|cold|canterbury" test.txt 3. Apply patch and restart services 4. Repeat step 2 and notice 2 of the URLs are now returning 200 OK status codes. Sponsored-by: Earth Sciences New Zealand -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30614 --- Comment #8 from David Cook <dcook@prosentient.com.au> --- (In reply to Aleisha Amohia from comment #7)
This enhancement makes a GET request as a fallback if HEAD returns an error status.
This doesn't fix all of the errors produced by the sample error MARC attached - any further support is welcomed.
It's a challenging one. On one hand, we're trying to stop people from using bots against Koha. On the other hand, we'd like to use bots to do things like link checking in Koha. We've had a custom link checker for many years that sits adjacent to Koha, and what I did create a hash of configurable domains and then check each link checker URL against that list first. If it matches, then we don't bother checking it, because we know the site is just going to block our link checker attempt anyway. Obviously, it means your link checking is never going to be perfect. But it makes for fewer false positives. To get the more accurate result, we'd have to use a headless browser set up to pretend to be a real human, but then we'd also have become the thing that we were trying to protect ourselves against. Of course, I think we could argue our purposes are more positive, but I don't know that the link checker targets would necessarily agree. Anyway, that's a lot of text for a small idea. That's what I ended up doing locally (outside of Koha) to deal with this sort of situation. Not perfect but it's been fairly practical. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30614 --- Comment #9 from Aleisha Amohia <aleisha@catalyst.net.nz> --- (In reply to David Cook from comment #8)
(In reply to Aleisha Amohia from comment #7)
This enhancement makes a GET request as a fallback if HEAD returns an error status.
This doesn't fix all of the errors produced by the sample error MARC attached - any further support is welcomed.
It's a challenging one. On one hand, we're trying to stop people from using bots against Koha. On the other hand, we'd like to use bots to do things like link checking in Koha.
We've had a custom link checker for many years that sits adjacent to Koha, and what I did create a hash of configurable domains and then check each link checker URL against that list first. If it matches, then we don't bother checking it, because we know the site is just going to block our link checker attempt anyway.
Obviously, it means your link checking is never going to be perfect. But it makes for fewer false positives.
To get the more accurate result, we'd have to use a headless browser set up to pretend to be a real human, but then we'd also have become the thing that we were trying to protect ourselves against. Of course, I think we could argue our purposes are more positive, but I don't know that the link checker targets would necessarily agree.
Anyway, that's a lot of text for a small idea. That's what I ended up doing locally (outside of Koha) to deal with this sort of situation. Not perfect but it's been fairly practical.
I totally hear you. The custom link checker which ignores allowlisted domains is a reasonable workaround, just trying to avoid custom as much as possible these days! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30614 --- Comment #10 from David Cook <dcook@prosentient.com.au> --- (In reply to Aleisha Amohia from comment #9)
I totally hear you. The custom link checker which ignores allowlisted domains is a reasonable workaround, just trying to avoid custom as much as possible these days!
Yep! That makes a lot of sense. I'm trying to do the same. What I meant was maybe a way to enhance the check-quick-url would be to add a patch for that via bugzilla. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30614 --- Comment #11 from David Cook <dcook@prosentient.com.au> --- Comment on attachment 187030 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=187030 Bug 30614: Fallback to a GET request if HEAD returns error status Review of attachment 187030: --> (https://bugs.koha-community.org/bugzilla3/page.cgi?id=splinter.html&bug=30614&attachment=187030) ----------------------------------------------------------------- ::: misc/cronjobs/check-url-quick.pl @@ +115,5 @@
+ my $request = HTTP::Request->new( 'GET' => $url ); + my $ua = LWP::UserAgent->new; + $ua->agent($user_agent); + my $response = $ua->request($request); + if ( $response->is_redirect ) {
I think that LWP::UserAgent follows redirects by default, so I don't think this code would be triggered? For instance, if you curl https://google.com you'll get a 301 but if you use LWP::UserAgent with that HTTP::Request GET, you'll get a 200. It's not clear to me at a glance if AnyEvent::HTTP::http_request follows redirects by default or not. At a glance, it looks like maybe not. @@ +120,5 @@
+ my $redirect_url = $response->header('Location'); + $redirect_url = URI->new_abs( $redirect_url, $url ); + $response = $ua->get($redirect_url); + } + my $status_code = substr( $response->status_line, 0, 3 );
You can just use $response->code() here. @@ +121,5 @@
+ $redirect_url = URI->new_abs( $redirect_url, $url ); + $response = $ua->get($redirect_url); + } + my $status_code = substr( $response->status_line, 0, 3 ); + my $reason = substr( $response->status_line, 3 );
You can use $response->message() here. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30614 --- Comment #12 from Aleisha Amohia <aleisha@catalyst.net.nz> --- (In reply to David Cook from comment #10)
(In reply to Aleisha Amohia from comment #9)
I totally hear you. The custom link checker which ignores allowlisted domains is a reasonable workaround, just trying to avoid custom as much as possible these days!
Yep! That makes a lot of sense. I'm trying to do the same.
What I meant was maybe a way to enhance the check-quick-url would be to add a patch for that via bugzilla.
Absolutely - I've suggested a syspref to the library where they could configure their allowlisted domains, will see if that's a suitable solution for them. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30614 --- Comment #13 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 188300 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=188300&action=edit Bug 30614: [alternate] Fallback to a GET request if HEAD returns error status This enhancement makes a GET request as a fallback if HEAD returns an error status using the existing AnyEvent module to maintain async checking of URLs. This doesn't fix all of the errors produced by the sample error MARC attached - any further support is welcomed. This patch adds a --where option to limit the biblios checked, and an extra level of verbosity to see the full response To test: 1. Download MARC. Go to Cataloguing in the staff interface and stage the MARC file for import, then import. 2. Run the URL check job and confirm erroring URLs perl misc/cronjobs/check-url-quick.pl -v > test.txt grep -E "tand|wiley|natlib|cold|canterbury" test.txt 3. Apply patch and restart services 4. Repeat step 2 and notice 2 of the URLs are now returning 200 OK status codes. Sponsored-by: Earth Sciences New Zealand -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30614 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nick@bywatersolutions.com --- Comment #14 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- I think we use AnyEvent because it is non-blocking, and we can send a bunch of requests at once. It does seem to handle redirects: https://metacpan.org/pod/AnyEvent::HTTP I added a where and verbose option to make it easier to test. Running like this perl misc/cronjobs/check-url-quick.pl -v -v --where="biblionumber>435" > test.txt I found that the full responses contain 'cf-mitigated' => 'challenge' so I think the remaining issue is that we are being blocked as a bot by CloudFlare, I don't know how to work around that -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30614 --- Comment #15 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- @ping Aleisha What do you think about Nick's alternative approach? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30614 --- Comment #16 from Aleisha Amohia <aleisha@catalyst.net.nz> --- (In reply to Katrin Fischer from comment #15)
@ping Aleisha What do you think about Nick's alternative approach?
I'm happy with whatever approach works! It looks good to me -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30614 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david@davidnind.com Attachment #187030|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30614 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off Comma delimited| |Earth Sciences New Zealand list of Sponsors| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30614 --- Comment #17 from David Nind <david@davidnind.com> --- Created attachment 200873 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200873&action=edit Bug 30614: Fallback to a GET request if HEAD returns error status This enhancement makes a GET request as a fallback if HEAD returns an error status using the existing AnyEvent module to maintain async checking of URLs. This doesn't fix all of the errors produced by the sample error MARC attached - any further support is welcomed. This patch adds a --where option to limit the biblios checked, and an extra level of verbosity to see the full response To test: 1. Download MARC. Go to Cataloguing in the staff interface and stage the MARC file for import, then import. 2. Run the URL check job and confirm erroring URLs perl misc/cronjobs/check-url-quick.pl -v > test.txt grep -E "tand|wiley|natlib|cold|canterbury" test.txt 3. Apply patch and restart services 4. Repeat step 2 and notice 2 of the URLs are now returning 200 OK status codes. Sponsored-by: Earth Sciences New Zealand Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30614 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #188300|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30614 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version|20.05 |Main --- Comment #18 from David Nind <david@davidnind.com> --- I renamed the alternate patch and obsoleted the original patch. Testing notes (using KTD): 1. Before the patch: grep -E "tand|wiley|natlib|cold|canterbury" test.txt 441 https://ir.canterbury.ac.nz/handle/10092/1507 200 OK 440 http://onlinelibrary.wiley.com/journal/10.1002/%28ISSN%291552-8618/issues 200 OK 442 https://natlib-primo.hosted.exlibrisgroup.com/permalink/f/1s57t7d/NLNZ_ALMA1... 403 Cache-Control: private 443 https://coldregions.americangeosciences.org/vufind/Content/ajus 404 Not Found 439 http://www.tandfonline.com/loi/tejp20 200 OK 2. After the alternative patch (I deleted the test.txt file before running again): grep -E "tand|wiley|natlib|cold|canterbury" test.txt 443 https://coldregions.americangeosciences.org/vufind/Content/ajus 200 OK 442 https://natlib-primo.hosted.exlibrisgroup.com/permalink/f/1s57t7d/NLNZ_ALMA1... 302 Location: /primo_library/libweb/permalink?state=f&key=1s57t7d&docid=NLNZ_ALMA11271200280002836 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30614 --- Comment #19 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- You beat me to it, David :) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30614 --- Comment #20 from David Nind <david@davidnind.com> --- (In reply to Katrin Fischer from comment #19)
You beat me to it, David :)
8-; -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org