[Bug 13568] New: OAI Server doesn't handle properly resumption token
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13568 Bug ID: 13568 Summary: OAI Server doesn't handle properly resumption token Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Web services Assignee: koha-bugs@lists.koha-community.org Reporter: frederic@tamil.fr QA Contact: testopia@bugs.koha-community.org When responding to ListRecords and ListIdentifiers verbs, OAI server doesn't return proper resumption token. At the end of a result set, OAI server generates a resumption token even if there isn't anymore records. Consequently, OAI harverster will send a new request, based on this invalid resumption, token. OAI Server responds with an empty resultset, which is considered as an invalid response by most of the harvesters. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13568 Frédéric Demians <frederic@tamil.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |frederic@tamil.fr |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13568 Frédéric Demians <frederic@tamil.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff Patch complexity|--- |Small patch -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13568 --- Comment #1 from Frédéric Demians <frederic@tamil.fr> --- Created attachment 35150 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=35150&action=edit Bug 13568 OAI Server doesn't handle properly resumption token When responding to ListRecords and ListIdentifiers verbs, OAI server doesn't return proper resumption token. At the end of a result set, OAI server generates a resumption token even if there isn't anymore records. Consequently, OAI harverster will send a new request, based on this invalid resumption, token. OAI Server responds with an empty resultset, which is considered as an invalid response by most of the harvesters. TO TEST: - Find in your DB, a day where a few biblio records have been created. The number of created biblios must inferior to OAI-PMH:MaxCount. - Let say this day is 2014-01-09. Send an OAI-PMH request to Koha OAI Server: /cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=marcxml&from=2014-01-09&until=2014-01-09 - At the end of the result, you will see a resumption token which looks like that: <resumptionToken cursor="47">marcxml/47/2014-01-09/2014-01-09/</resumptionToken> This is wrong. No resumptiion token should be sent since there isn't anymore records to harvest. - Apply the patch. - Resend the OAI-PMH request. There is no resumption token at the end of the result. - You could test also with ListIdenfiers verb in place of ListRecord. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13568 Christophe Brocquet <christophe.brocquet@yahoo.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |christophe.brocquet@yahoo.f | |r --- Comment #2 from Christophe Brocquet <christophe.brocquet@yahoo.fr> --- I tested the patch for the 13568 with the following sandbox http://pro.test1.biblibre.com/cgi-bin/koha/sandbox.pl I used the "UNIMARC public library (2) = UNIMARC BM" dataset. 1. First I activated the following system preferences in the staff interface - OAI-PMH : enabled - OAI-PMH:AutoUpdateSets : enabled - OAI-PMH:MaxCount : 500 2. Then I tested with - http://catalogue.test1.biblibre.com/cgi-bin/koha/oai.pl/request?verb=ListIdentifiers&metadataPrefix=marcxml - http://catalogue.test1.biblibre.com/cgi-bin/koha/oai.pl/request?verb=ListRecords&metadataPrefix=marcxml 3. The last resumtionToken which was generated is "marcxml/4500/1970-01-01/2015-02-17/" and the last identifier was "KOHA-OAI-TEST:5000" : after this record there was no more resumptionToken. But it is said here http://koha-fr.org/content/bacs-%C3%A0-sable that the "UNIMARC public library (2) = UNIMARC BM" dataset contains 19000 bibliographical records. So I am not sure the patch works well. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13568 --- Comment #3 from Frédéric Demians <frederic@tamil.fr> ---
3. The last resumtionToken which was generated is "marcxml/4500/1970-01-01/2015-02-17/" and the last identifier was "KOHA-OAI-TEST:5000" : after this record there was no more resumptionToken. But it is said here http://koha-fr.org/content/bacs-%C3%A0-sable that the "UNIMARC public library (2) = UNIMARC BM" dataset contains 19000 bibliographical records.
So I am not sure the patch works well.
You could see that without the patch, even asking records with "marcxml/4500/1970-01-01/2015-02-17/" resumption token, you will have a resumption token in the response. The dataset doesn't contain 19000 records but 4848. You can check with this SQL query: SELECT COUNT(*) FROM biblio. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13568 sandboxes@biblibre.com <sandboxes@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off CC| |sandboxes@biblibre.com --- Comment #4 from sandboxes@biblibre.com <sandboxes@biblibre.com> --- Patch tested with a sandbox, by Christophe Brocquet <christophe.brocquet@obspm.fr> -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13568 sandboxes@biblibre.com <sandboxes@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #35150|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13568 --- Comment #5 from sandboxes@biblibre.com <sandboxes@biblibre.com> --- Created attachment 35962 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=35962&action=edit Bug 13568 OAI Server doesn't handle properly resumption token When responding to ListRecords and ListIdentifiers verbs, OAI server doesn't return proper resumption token. At the end of a result set, OAI server generates a resumption token even if there isn't anymore records. Consequently, OAI harverster will send a new request, based on this invalid resumption, token. OAI Server responds with an empty resultset, which is considered as an invalid response by most of the harvesters. TO TEST: - Find in your DB, a day where a few biblio records have been created. The number of created biblios must inferior to OAI-PMH:MaxCount. - Let say this day is 2014-01-09. Send an OAI-PMH request to Koha OAI Server: /cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=marcxml&from=2014-01-09&until=2014-01-09 - At the end of the result, you will see a resumption token which looks like that: <resumptionToken cursor="47">marcxml/47/2014-01-09/2014-01-09/</resumptionToken> This is wrong. No resumptiion token should be sent since there isn't anymore records to harvest. - Apply the patch. - Resend the OAI-PMH request. There is no resumption token at the end of the result. - You could test also with ListIdenfiers verb in place of ListRecord. Signed-off-by: Christophe Brocquet <christophe.brocquet@obspm.fr> -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13568 Jonathan Druart <jonathan.druart@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@biblibre.co | |m --- Comment #6 from Jonathan Druart <jonathan.druart@biblibre.com> --- (In reply to Frédéric Demians from comment #3)
The dataset doesn't contain 19000 records but 4848. You can check with this SQL query: SELECT COUNT(*) FROM biblio.
mysql> select count(*) from biblio; +----------+ | count(*) | +----------+ | 4848 | +----------+ 1 row in set (0.00 sec) -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13568 Jonathan Druart <jonathan.druart@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13568 Jonathan Druart <jonathan.druart@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #35962|0 |1 is obsolete| | --- Comment #7 from Jonathan Druart <jonathan.druart@biblibre.com> --- Created attachment 36018 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=36018&action=edit Bug 13568 OAI Server doesn't handle properly resumption token When responding to ListRecords and ListIdentifiers verbs, OAI server doesn't return proper resumption token. At the end of a result set, OAI server generates a resumption token even if there isn't anymore records. Consequently, OAI harverster will send a new request, based on this invalid resumption, token. OAI Server responds with an empty resultset, which is considered as an invalid response by most of the harvesters. TO TEST: - Find in your DB, a day where a few biblio records have been created. The number of created biblios must inferior to OAI-PMH:MaxCount. - Let say this day is 2014-01-09. Send an OAI-PMH request to Koha OAI Server: /cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=marcxml&from=2014-01-09&until=2014-01-09 - At the end of the result, you will see a resumption token which looks like that: <resumptionToken cursor="47">marcxml/47/2014-01-09/2014-01-09/</resumptionToken> This is wrong. No resumptiion token should be sent since there isn't anymore records to harvest. - Apply the patch. - Resend the OAI-PMH request. There is no resumption token at the end of the result. - You could test also with ListIdenfiers verb in place of ListRecord. Signed-off-by: Christophe Brocquet <christophe.brocquet@obspm.fr> Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13568 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to Master CC| |tomascohen@gmail.com --- Comment #8 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Patch pushed to master. Thanks Frederic! -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13568 Chris Cormack <chris@bigballofwax.co.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to Master |Pushed to Stable CC| |chris@bigballofwax.co.nz --- Comment #9 from Chris Cormack <chris@bigballofwax.co.nz> --- No string changes pushed to 3.18.x will be in 3.18.4 -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13568 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au --- Comment #10 from David Cook <dcook@prosentient.com.au> --- Hmm, I haven't had any issues with the resumption tokens... I'll have to take a gander at this sometime... -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13568 --- Comment #11 from Frédéric Demians <frederic@tamil.fr> --- (In reply to David Cook from comment #10)
Hmm, I haven't had any issues with the resumption tokens... I'll have to take a gander at this sometime...
It depends on OAI-PMH harvester: some accepts an empty result set returned to a resumption query. A well known discovery tool, edited by an US-Israeli company, fails to handle an empty resultset, report it, and stop harvesting. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13568 Mason James <mtj@kohaaloha.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |rel_3_16_candidate CC| |mtj@kohaaloha.com --- Comment #12 from Mason James <mtj@kohaaloha.com> --- Possible selection for 3.16.x -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13568 Mason James <mtj@kohaaloha.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|rel_3_16_candidate | --- Comment #13 from Mason James <mtj@kohaaloha.com> --- Pushed to 3.16.x, will be in 3.16.10 -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13568 Frédéric Demians <frederic@tamil.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |12252 Referenced Bugs: http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12252 [Bug 12252] OAI-PMH GetRecord result doesn't include item data -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13568 Gaetan Boisson <gaetan.boisson@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |12252 Depends on|12252 | Referenced Bugs: http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12252 [Bug 12252] OAI-PMH GetRecord result doesn't include item data -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13568 Fridolin SOMERS <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m Resolution|--- |FIXED Status|Pushed to Stable |RESOLVED --- Comment #14 from Fridolin SOMERS <fridolin.somers@biblibre.com> --- This patch will not be in 3.14.x so set as resolved -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org