[Bug 10974] New: OAI-PMH Resumption Tokens Do Not Handle Time
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10974 Bug ID: 10974 Summary: OAI-PMH Resumption Tokens Do Not Handle Time Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Web services Assignee: koha-bugs@lists.koha-community.org Reporter: dcook@prosentient.com.au QA Contact: testopia@bugs.koha-community.org Currently, it is possible to retrieve 50 or so records from Koha via OAI-PMH, when using times in your 'from' and 'until' arguments. Here is an example that will return a list of records (if your OAI server syspref is turned on). http://KOHAINSTANCE/cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=oai_dc&from=2012-09-05T13:44:33Z&until=2014-09-05T13:44:33Z However, if you need to retrieve the rest of the results via a resumption token, Koha won't return your results. http://KOHAINSTANCE/cgi-bin/koha/oai.pl?verb=ListRecords&resumptionToken=oai_dc:50:2012-09-05T13:44:33Z:2014-09-05T13:44:33Z: This is because Koha joins and splits the resumption token using colons. Obviously, since the timestamps use colons, the string isn't going to be split correctly. I propose that we change the separator colons to slashes. This is the method that DSpace uses when serving records via OAI-PMH. You can see an example here: http://papyrus.bib.umontreal.ca/dspace-oai/request?verb=ListRecords&metadataPrefix=oai_dc -- Note: Koha doesn't handle times 100% correctly in master either. That's why I opened bug 10824. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10974 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |http://bugs.koha-community. | |org/bugzilla3/show_bug.cgi? | |id=10824 -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10974 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |dcook@prosentient.com.au |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10974 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff Patch complexity|--- |Small patch -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10974 --- Comment #1 from David Cook <dcook@prosentient.com.au> --- Created attachment 21633 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=21633&action=edit Bug 10974 - OAI-PMH Resumption Tokens Do Not Handle Time This patch changes the value separator in OAI-PMH resumption tokens from colons to slashes, so that the token string isn't split incorrectly when a time is included. TEST PLAN: 1) Turn on the OAI-PMH server syspref in Koha 2) Send a ListRecords request using 'from' and 'until' arguments that include times (Best to use very far apart times so that you retrieve more than 50 records which will likely be the trigger for a resumptionToken). Here is an example: http://KOHAINSTANCE/cgi-bin/koha/oai.pl?verb=ListRecords& metadataPrefix=oai_dc&from=2012-09-05T13:44:33Z&until=2014-09-05T13:44:33Z N.B. Replace KOHAINSTANCE with the URL of your Koha instance. 3) Scroll down to the bottom of the page until you find the resumptionToken. It will look similar to this: <resumptionToken cursor="50"> oai_dc:50:2012-09-05T13:44:33Z:2014-09-05T13:44:33Z: </resumptionToken> 4) Copy that resumption token and send a request with it like so: http://KOHAINSTANCE/cgi-bin/koha/oai.pl?verb=ListRecords& resumptionToken=oai_dc:50:2012-09-05T13:44:33Z:2014-09-05T13:44:33Z: 5) The page should (incorrectly) show no records. 6) APPLY PATCH 7) Repeat steps 2, 3, and 4 8) Note that the resumptionToken now uses slashes (e.g. /) instead of colons. Note also that now the second request will show records!!! N.B. This will only happen if Koha has enough records to serve to you. If your Koha has less than 50 records, try lowering the number provided in the "OAI-PMH:MaxCount" system preference. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10974 I'm just a bot <gitbot@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |gitbot@bugs.koha-community. | |org When did the bot| |2013-10-02 last check this| | --- Comment #2 from I'm just a bot <gitbot@bugs.koha-community.org> --- Patch applied cleanly, go forth and signoff -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10974 Petter Goksøyr Åsen <boutrosboutrosboutros@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |boutrosboutrosboutros@gmail | |.com --- Comment #3 from Petter Goksøyr Åsen <boutrosboutrosboutros@gmail.com> --- Strange, the resumptionToken request works for me, without applying the patch. It also works after I apply the patch of course. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10974 --- Comment #4 from David Cook <dcook@prosentient.com.au> --- (In reply to Petter Goksøyr Åsen from comment #3)
Strange, the resumptionToken request works for me, without applying the patch.
It also works after I apply the patch of course.
Thanks for that Petter. I should've written my test plan better. In fact, it only looks like it's working. If you insert some debugging code, you will find that the following token: oai_dc:50:2012-09-05T13:44:33Z:2013-09-05T13:44:33Z: is being broken into the following: metadataPrefix = oai_dc offset = 50 from = 2012-09-05T13 until = 44 set = 33Z and the rest of the token will be thrown away. -- If you re-run the test, take a look at the first result you receive in the ListRecords response. Change your "until" date to before the datestamp of that first result. You will find that record still appears when it shouldn't! -- Also, the resumption token at the bottom will be similar to the following: oai_dc:66:2012-09-05T13:44:33Z This looks like a resumption token with a fully formed From date. However, it's a token built up from those split fields I identified before. It thinks 44 is the until datestamp and 33Z is the set. If you try to use that resumptionToken, you will get no results. -- Please try it again in light of these notes. I'm sure that you'll find I'm correct :) -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10974 Petter Goksøyr Åsen <boutrosboutrosboutros@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10974 Petter Goksøyr Åsen <boutrosboutrosboutros@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #21633|0 |1 is obsolete| | --- Comment #5 from Petter Goksøyr Åsen <boutrosboutrosboutros@gmail.com> --- Created attachment 22239 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=22239&action=edit Bug 10974 - OAI-PMH Resumption Tokens Do Not Handle Time This patch changes the value separator in OAI-PMH resumption tokens from colons to slashes, so that the token string isn't split incorrectly when a time is included. TEST PLAN: 1) Turn on the OAI-PMH server syspref in Koha 2) Send a ListRecords request using 'from' and 'until' arguments that include times (Best to use very far apart times so that you retrieve more than 50 records which will likely be the trigger for a resumptionToken). Here is an example: http://KOHAINSTANCE/cgi-bin/koha/oai.pl?verb=ListRecords& metadataPrefix=oai_dc&from=2012-09-05T13:44:33Z&until=2014-09-05T13:44:33Z N.B. Replace KOHAINSTANCE with the URL of your Koha instance. 3) Scroll down to the bottom of the page until you find the resumptionToken. It will look similar to this: <resumptionToken cursor="50"> oai_dc:50:2012-09-05T13:44:33Z:2014-09-05T13:44:33Z: </resumptionToken> 4) Copy that resumption token and send a request with it like so: http://KOHAINSTANCE/cgi-bin/koha/oai.pl?verb=ListRecords& resumptionToken=oai_dc:50:2012-09-05T13:44:33Z:2014-09-05T13:44:33Z: 5) The page should (incorrectly) show no records. 6) APPLY PATCH 7) Repeat steps 2, 3, and 4 8) Note that the resumptionToken now uses slashes (e.g. /) instead of colons. Note also that now the second request will show records!!! N.B. This will only happen if Koha has enough records to serve to you. If your Koha has less than 50 records, try lowering the number provided in the "OAI-PMH:MaxCount" system preference. Signed-off-by: Petter Goksoyr Asen <boutrosboutrosboutros@gmail.com> I understand; I can now confirm the behaviour described in the test plan. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10974 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10974 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #22239|0 |1 is obsolete| | --- Comment #6 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 22450 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=22450&action=edit [PASSED QA] Bug 10974 - OAI-PMH Resumption Tokens Do Not Handle Time This patch changes the value separator in OAI-PMH resumption tokens from colons to slashes, so that the token string isn't split incorrectly when a time is included. TEST PLAN: 1) Turn on the OAI-PMH server syspref in Koha 2) Send a ListRecords request using 'from' and 'until' arguments that include times (Best to use very far apart times so that you retrieve more than 50 records which will likely be the trigger for a resumptionToken). Here is an example: http://KOHAINSTANCE/cgi-bin/koha/oai.pl?verb=ListRecords& metadataPrefix=oai_dc&from=2012-09-05T13:44:33Z&until=2014-09-05T13:44:33Z N.B. Replace KOHAINSTANCE with the URL of your Koha instance. 3) Scroll down to the bottom of the page until you find the resumptionToken. It will look similar to this: <resumptionToken cursor="50"> oai_dc:50:2012-09-05T13:44:33Z:2014-09-05T13:44:33Z: </resumptionToken> 4) Copy that resumption token and send a request with it like so: http://KOHAINSTANCE/cgi-bin/koha/oai.pl?verb=ListRecords& resumptionToken=oai_dc:50:2012-09-05T13:44:33Z:2014-09-05T13:44:33Z: 5) The page should (incorrectly) show no records. 6) APPLY PATCH 7) Repeat steps 2, 3, and 4 8) Note that the resumptionToken now uses slashes (e.g. /) instead of colons. Note also that now the second request will show records!!! N.B. This will only happen if Koha has enough records to serve to you. If your Koha has less than 50 records, try lowering the number provided in the "OAI-PMH:MaxCount" system preference. Signed-off-by: Petter Goksoyr Asen <boutrosboutrosboutros@gmail.com> I understand; I can now confirm the behaviour described in the test plan. Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de> Passes test plan, all tests and QA script. Resumption Token works correctly after applying the patch. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10974 Galen Charlton <gmcharlt@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to Master CC| |gmcharlt@gmail.com --- Comment #7 from Galen Charlton <gmcharlt@gmail.com> --- Pushed to master. Thanks, David! -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10974 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to Master |Pushed to Stable CC| |tomascohen@gmail.com --- Comment #8 from Tomás Cohen Arazi <tomascohen@gmail.com> --- This patch has been pushed to 3.12.x, will be in 3.12.8. Thanks David! -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org