[Bug 42952] New: Public SRU with OPAC logic out-of-the-box
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 Bug ID: 42952 Summary: Public SRU with OPAC logic out-of-the-box Initiative type: --- Sponsorship --- status: Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: new feature Priority: P5 - low Component: Z39.50 / SRU / OpenSearch Servers Assignee: koha-bugs@lists.koha-community.org Reporter: dcook@prosentient.com.au QA Contact: testopia@bugs.koha-community.org CC: m.de.rooy@rijksmuseum.nl Target Milestone: --- While working with Terry Reese to fix the MarcEdit integration with Koha, he pointed out that Koha doesn't have a public Z39.50/SRU endpoint out of the box. I think the reasoning previously has been that the z3950responder.pl and Zebra provide unfiltered access to the indexes. But... with the z3950responder.pl we could provide a more mediated access. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 Mathieu Saby <mathsabypro@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mathsabypro@gmail.com --- Comment #1 from Mathieu Saby <mathsabypro@gmail.com> --- Maybe duplicate of https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40700 ? -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 --- Comment #2 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Mathieu Saby from comment #1)
Maybe duplicate of https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40700 ?
I think this is maybe one possible complication. Maybe it should still be a switch, but something the library can activate from the GUI? We have 2 different SRU server implementations currently and you might not want to offer both in parallel (Zebra and Elasticsearch). Do OpacSuppression, Hiding of MARC fields via visibility settings, OpacHiddenItems all work with SRU? I could see these as a reason why a library might not want to make it publicly available. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 --- Comment #3 from David Cook <dcook@prosentient.com.au> --- (In reply to Katrin Fischer from comment #2)
(In reply to Mathieu Saby from comment #1)
Maybe duplicate of https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40700 ?
Possibly although I think you're misunderstanding the role of SRU with Koha in bug 40700.
I think this is maybe one possible complication. Maybe it should still be a switch, but something the library can activate from the GUI?
100%
We have 2 different SRU server implementations currently and you might not want to offer both in parallel (Zebra and Elasticsearch).
It's a bit more complex than that. The server implementations are Zebra or z3950responder.pl. (Zebra being configured through the horribly named koha-conf.xml) Now z3950responder.pl can use Zebra or Elastic as a backend. It chooses its backend based off the "SearchEngine" system preference. Note that z3950responder.pl doesn't support authentication while Zebra does support authentication.
Do OpacSuppression, Hiding of MARC fields via visibility settings, OpacHiddenItems all work with SRU?
When using Zebra, it's pure Zebra and no Koha logic, so no those are not supported. When using z3950responder.pl with a Zebra backend, the query is passed straight through to Zebra, so no those are not supported. When using z3950responder.pl with an Elasticsearch backend, the search is mediated by Koha::SearchEngine::* modules, so there are some possibilities for support, but I think they don't do them by default.
I could see these as a reason why a library might not want to make it publicly available.
100% I think we'd want to use Unix sockets (for ease of use if nothing else) and then we'd also want to make sure we're honouring Koha's OPAC business logic (e.g. OpacSuppression, Hiding of MARC fields via visibility settings, OpacHiddenItems, etc) SRU is just a protocol. We can control what we send, although we'll run into the age old problems we have with OpacHiddenItems... -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=40700 -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 --- Comment #4 from David Cook <dcook@prosentient.com.au> --- @Katrin if we had a public SRU that honoured the OPAC business logic, we could potentially also look at that being used for a next-generation for the OPAC search interface. If loading opac-search.pl was actually super fast and initiated the search, we could have a lot more control over how searches happen. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 --- Comment #5 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Maybe with the known limitations it would be good if we added authentication as an option to the z3950_responder? -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 --- Comment #6 from David Cook <dcook@prosentient.com.au> --- (In reply to Katrin Fischer from comment #5)
Maybe with the known limitations it would be good if we added authentication as an option to the z3950_responder?
It would be good to add authentication to the z3950_responder.pl for sure -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |clemens.tubach@kit.edu, | |michaela.sieber@kit.edu -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 Mark Hofstetter <mark@hofstetter.at> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mark@hofstetter.at -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 --- Comment #7 from Mark Hofstetter <mark@hofstetter.at> --- customer wants SRU to respect OpacSuppression and "munging" of marcxml output ie suppress/select which sub/fields to show. will implement. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 --- Comment #8 from David Cook <dcook@prosentient.com.au> --- (In reply to Mark Hofstetter from comment #7)
customer wants SRU to respect OpacSuppression and "munging" of marcxml output ie suppress/select which sub/fields to show. will implement.
Cool. I'm happy to review. Are you planning to use the z3950responder.pl? I'm keen to get this one done, but it's been low on my priorities for coding it myself. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 --- Comment #9 from Mark Hofstetter <mark@hofstetter.at> --- Created attachment 206025 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=206025&action=edit Bug 42952: Add public SRU configuration Add opt-in OPAC visibility rules for bibliographic requests received by the shared Z39.50/SRU responder when Elasticsearch is used. Suppressed and OPAC-hidden records are not exposed. A direct numeric rec.id request returns a suppression diagnostic only when OpacSuppressionRedirect uses the blocked-record response; otherwise it returns no hit. Add PublicSRUXSLT for optional MARCXML transformations after OPAC MARC filtering. Include a demo stylesheet that returns only 245$a, and return a temporary diagnostic when a stylesheet fails or does not return MARCXML. Document the configuration and add tests for all preference, suppression, redirect, IP-range, OPAC-hidden-item, XSLT, and Elasticsearch cases. Test plan: 1. Apply the patch, run database updates, and use Elasticsearch. 2. Restart the responder after changing preferences: koha-z3950-responder --restart kohadev 3. Use visible biblio 1 and set biblio 191 to opac_suppressed. 4. Set PublicSRU to Use, OpacSuppression to Hide, clear OpacSuppressionByIPRange, set OpacSuppressionRedirect to the not-found setting, and leave PublicSRUXSLT empty. 5. Confirm that a visible biblio returns OPAC-filtered MARCXML: docker exec ktdmaster-koha-1 curl -sS -G 'http://127.0.0.1:2100/biblios' \ --data-urlencode 'version=1.1' \ --data-urlencode 'operation=searchRetrieve' \ --data-urlencode 'query=rec.id=1' \ --data-urlencode 'recordSchema=marcxml' \ --data-urlencode 'recordPacking=xml' \ --data-urlencode 'maximumRecords=1' 6. Confirm that a direct request for the suppressed biblio returns no hit and does not disclose suppression: docker exec ktdmaster-koha-1 curl -sS -G 'http://127.0.0.1:2100/biblios' \ --data-urlencode 'version=1.1' \ --data-urlencode 'operation=searchRetrieve' \ --data-urlencode 'query=rec.id=191' \ --data-urlencode 'recordSchema=marcxml' \ --data-urlencode 'recordPacking=xml' \ --data-urlencode 'maximumRecords=1' 7. Confirm that a normal search cannot disclose the suppressed biblio: docker exec ktdmaster-koha-1 curl -sS -G 'http://127.0.0.1:2100/biblios' \ --data-urlencode 'version=1.1' \ --data-urlencode 'operation=searchRetrieve' \ --data-urlencode 'query=rec.id=1 or rec.id=191' \ --data-urlencode 'recordSchema=marcxml' \ --data-urlencode 'recordPacking=xml' \ --data-urlencode 'maximumRecords=10' 8. Enable the blocked-record OpacSuppressionRedirect setting and repeat the direct request. It must contain Record is suppressed. The normal search must still hide biblio 191. 9. Set OpacSuppressionByIPRange to 127\\.0\\.0\\. and repeat the direct request from the container. It must return biblio 191. Disable OpacSuppression and confirm that the biblio also remains visible. 10. Set PublicSRUXSLT to /kohadevbox/koha/koha-tmpl/intranet-tmpl/prog/en/xslt/MARC21slim245a.xsl, restart the responder, and confirm that the visible biblio has only 245$a in its MARCXML. Verify an invalid stylesheet returns the temporary MARCXML transformation diagnostic. 11. Configure an OpacHiddenItems rule and a matching item. Confirm that it is not exposed, including from an IP-range exception. 12. Confirm the shared responder behaviour over Z39.50: printf '%s\n' 'base biblios' 'find @attr 1=12 191' 'show 1' 'quit' | \ docker exec -i ktdmaster-koha-1 yaz-client 127.0.0.1:2100 13. Run: prove -v t/Koha/Z3950Responder/PublicSRU.t prove -v t/db_dependent/Koha/Z3950Responder/GenericSession.t Co-authored-by: OpenAI Codex <codex@openai.com> -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 Mark Hofstetter <mark@hofstetter.at> changed: What |Removed |Added ---------------------------------------------------------------------------- Sponsorship status|--- |Sponsored Comma delimited| |Steiermärkische list of Sponsors| |Landesbibliothek -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 Mark Hofstetter <mark@hofstetter.at> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #206025|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 --- Comment #10 from Mark Hofstetter <mark@hofstetter.at> --- Created attachment 206026 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=206026&action=edit Bug 42952: Add public SRU configuration Add opt-in OPAC visibility rules for bibliographic requests received by the shared Z39.50/SRU responder when Elasticsearch is used. Suppressed and OPAC-hidden records are not exposed. A direct numeric rec.id request returns a suppression diagnostic only when OpacSuppressionRedirect uses the blocked-record response; otherwise it returns no hit. Add PublicSRUXSLT for optional MARCXML transformations after OPAC MARC filtering. Include a demo stylesheet that returns only 245$a, and return a temporary diagnostic when a stylesheet fails or does not return MARCXML. Document the configuration and add tests for all preference, suppression, redirect, IP-range, OPAC-hidden-item, XSLT, and Elasticsearch cases. Test plan: 1. Apply the patch, run database updates, and use Elasticsearch. 2. Restart the responder after changing preferences: koha-z3950-responder --restart kohadev 3. Use visible biblio 1 and set biblio 191 to opac_suppressed. 4. Set PublicSRU to Use, OpacSuppression to Hide, clear OpacSuppressionByIPRange, set OpacSuppressionRedirect to the not-found setting, and leave PublicSRUXSLT empty. 5. Confirm that a visible biblio returns OPAC-filtered MARCXML: docker exec ktdmaster-koha-1 curl -sS -G 'http://127.0.0.1:2100/biblios' \ --data-urlencode 'version=1.1' \ --data-urlencode 'operation=searchRetrieve' \ --data-urlencode 'query=rec.id=1' \ --data-urlencode 'recordSchema=marcxml' \ --data-urlencode 'recordPacking=xml' \ --data-urlencode 'maximumRecords=1' 6. Confirm that a direct request for the suppressed biblio returns no hit and does not disclose suppression: docker exec ktdmaster-koha-1 curl -sS -G 'http://127.0.0.1:2100/biblios' \ --data-urlencode 'version=1.1' \ --data-urlencode 'operation=searchRetrieve' \ --data-urlencode 'query=rec.id=191' \ --data-urlencode 'recordSchema=marcxml' \ --data-urlencode 'recordPacking=xml' \ --data-urlencode 'maximumRecords=1' 7. Confirm that a normal search cannot disclose the suppressed biblio: docker exec ktdmaster-koha-1 curl -sS -G 'http://127.0.0.1:2100/biblios' \ --data-urlencode 'version=1.1' \ --data-urlencode 'operation=searchRetrieve' \ --data-urlencode 'query=rec.id=1 or rec.id=191' \ --data-urlencode 'recordSchema=marcxml' \ --data-urlencode 'recordPacking=xml' \ --data-urlencode 'maximumRecords=10' 8. Enable the blocked-record OpacSuppressionRedirect setting and repeat the direct request. It must contain Record is suppressed. The normal search must still hide biblio 191. 9. Set OpacSuppressionByIPRange to 127\\.0\\.0\\. and repeat the direct request from the container. It must return biblio 191. Disable OpacSuppression and confirm that the biblio also remains visible. 10. Set PublicSRUXSLT to /kohadevbox/koha/koha-tmpl/intranet-tmpl/prog/en/xslt/MARC21slim245a.xsl, restart the responder, and confirm that the visible biblio has only 245$a in its MARCXML. Verify an invalid stylesheet returns the temporary MARCXML transformation diagnostic. 11. Configure an OpacHiddenItems rule and a matching item. Confirm that it is not exposed, including from an IP-range exception. 12. Confirm the shared responder behaviour over Z39.50: printf '%s\n' 'base biblios' 'find @attr 1=12 191' 'show 1' 'quit' | \ docker exec -i ktdmaster-koha-1 yaz-client 127.0.0.1:2100 13. Run: prove -v t/Koha/Z3950Responder/PublicSRU.t prove -v t/db_dependent/Koha/Z3950Responder/GenericSession.t Co-authored-by: OpenAI Codex <codex@openai.com> Sponsored-by: Steiermärkische Landesbibliothek -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 --- Comment #11 from Mark Hofstetter <mark@hofstetter.at> --- I didnt add Authentication, because "this should be done properly" (tm) using koha users + permissions which would put a little bit too much into one issue. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 --- Comment #12 from David Cook <dcook@prosentient.com.au> --- (In reply to Mark Hofstetter from comment #11)
I didnt add Authentication, because "this should be done properly" (tm) using koha users + permissions which would put a little bit too much into one issue.
Yeah, I also don't think we really need it for public SRU anyway at this stage. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |In Discussion Assignee|koha-bugs@lists.koha-commun |mark@hofstetter.at |ity.org | --- Comment #13 from David Cook <dcook@prosentient.com.au> --- Just moving this to "In Discussion" since you haven't marked it as Needs Signoff. Did you mean to put it into "Needs Signoff"? If so, I'd mark it as Failed QA because of some security issues. I'll add you to those, so you can see what changes will need to be made for this. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 --- Comment #14 from David Cook <dcook@prosentient.com.au> --- Really excited to see this work though. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 --- Comment #15 from David Cook <dcook@prosentient.com.au> --- Ah, also... did you add the XSLT so that they could control which fields are included? I think what you really want here is the Koha::RecordProcessor, as that's what applies the MARC Bibliographic Framework visibility rules. -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org