[Bug 42952] New: Public SRU with OPAC logic out-of-the-box
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 Bug ID: 42952 Summary: Public SRU with OPAC logic out-of-the-box Initiative type: --- Sponsorship --- status: Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: new feature Priority: P5 - low Component: Z39.50 / SRU / OpenSearch Servers Assignee: koha-bugs@lists.koha-community.org Reporter: dcook@prosentient.com.au QA Contact: testopia@bugs.koha-community.org CC: m.de.rooy@rijksmuseum.nl Target Milestone: --- While working with Terry Reese to fix the MarcEdit integration with Koha, he pointed out that Koha doesn't have a public Z39.50/SRU endpoint out of the box. I think the reasoning previously has been that the z3950responder.pl and Zebra provide unfiltered access to the indexes. But... with the z3950responder.pl we could provide a more mediated access. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 Mathieu Saby <mathsabypro@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mathsabypro@gmail.com --- Comment #1 from Mathieu Saby <mathsabypro@gmail.com> --- Maybe duplicate of https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40700 ? -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 --- Comment #2 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Mathieu Saby from comment #1)
Maybe duplicate of https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40700 ?
I think this is maybe one possible complication. Maybe it should still be a switch, but something the library can activate from the GUI? We have 2 different SRU server implementations currently and you might not want to offer both in parallel (Zebra and Elasticsearch). Do OpacSuppression, Hiding of MARC fields via visibility settings, OpacHiddenItems all work with SRU? I could see these as a reason why a library might not want to make it publicly available. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 --- Comment #3 from David Cook <dcook@prosentient.com.au> --- (In reply to Katrin Fischer from comment #2)
(In reply to Mathieu Saby from comment #1)
Maybe duplicate of https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40700 ?
Possibly although I think you're misunderstanding the role of SRU with Koha in bug 40700.
I think this is maybe one possible complication. Maybe it should still be a switch, but something the library can activate from the GUI?
100%
We have 2 different SRU server implementations currently and you might not want to offer both in parallel (Zebra and Elasticsearch).
It's a bit more complex than that. The server implementations are Zebra or z3950responder.pl. (Zebra being configured through the horribly named koha-conf.xml) Now z3950responder.pl can use Zebra or Elastic as a backend. It chooses its backend based off the "SearchEngine" system preference. Note that z3950responder.pl doesn't support authentication while Zebra does support authentication.
Do OpacSuppression, Hiding of MARC fields via visibility settings, OpacHiddenItems all work with SRU?
When using Zebra, it's pure Zebra and no Koha logic, so no those are not supported. When using z3950responder.pl with a Zebra backend, the query is passed straight through to Zebra, so no those are not supported. When using z3950responder.pl with an Elasticsearch backend, the search is mediated by Koha::SearchEngine::* modules, so there are some possibilities for support, but I think they don't do them by default.
I could see these as a reason why a library might not want to make it publicly available.
100% I think we'd want to use Unix sockets (for ease of use if nothing else) and then we'd also want to make sure we're honouring Koha's OPAC business logic (e.g. OpacSuppression, Hiding of MARC fields via visibility settings, OpacHiddenItems, etc) SRU is just a protocol. We can control what we send, although we'll run into the age old problems we have with OpacHiddenItems... -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=40700 -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 --- Comment #4 from David Cook <dcook@prosentient.com.au> --- @Katrin if we had a public SRU that honoured the OPAC business logic, we could potentially also look at that being used for a next-generation for the OPAC search interface. If loading opac-search.pl was actually super fast and initiated the search, we could have a lot more control over how searches happen. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 --- Comment #5 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Maybe with the known limitations it would be good if we added authentication as an option to the z3950_responder? -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 --- Comment #6 from David Cook <dcook@prosentient.com.au> --- (In reply to Katrin Fischer from comment #5)
Maybe with the known limitations it would be good if we added authentication as an option to the z3950_responder?
It would be good to add authentication to the z3950_responder.pl for sure -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |clemens.tubach@kit.edu, | |michaela.sieber@kit.edu -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 Mark Hofstetter <mark@hofstetter.at> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mark@hofstetter.at -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 --- Comment #7 from Mark Hofstetter <mark@hofstetter.at> --- customer wants SRU to respect OpacSuppression and "munging" of marcxml output ie suppress/select which sub/fields to show. will implement. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 --- Comment #8 from David Cook <dcook@prosentient.com.au> --- (In reply to Mark Hofstetter from comment #7)
customer wants SRU to respect OpacSuppression and "munging" of marcxml output ie suppress/select which sub/fields to show. will implement.
Cool. I'm happy to review. Are you planning to use the z3950responder.pl? I'm keen to get this one done, but it's been low on my priorities for coding it myself. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 --- Comment #9 from Mark Hofstetter <mark@hofstetter.at> --- Created attachment 206025 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=206025&action=edit Bug 42952: Add public SRU configuration Add opt-in OPAC visibility rules for bibliographic requests received by the shared Z39.50/SRU responder when Elasticsearch is used. Suppressed and OPAC-hidden records are not exposed. A direct numeric rec.id request returns a suppression diagnostic only when OpacSuppressionRedirect uses the blocked-record response; otherwise it returns no hit. Add PublicSRUXSLT for optional MARCXML transformations after OPAC MARC filtering. Include a demo stylesheet that returns only 245$a, and return a temporary diagnostic when a stylesheet fails or does not return MARCXML. Document the configuration and add tests for all preference, suppression, redirect, IP-range, OPAC-hidden-item, XSLT, and Elasticsearch cases. Test plan: 1. Apply the patch, run database updates, and use Elasticsearch. 2. Restart the responder after changing preferences: koha-z3950-responder --restart kohadev 3. Use visible biblio 1 and set biblio 191 to opac_suppressed. 4. Set PublicSRU to Use, OpacSuppression to Hide, clear OpacSuppressionByIPRange, set OpacSuppressionRedirect to the not-found setting, and leave PublicSRUXSLT empty. 5. Confirm that a visible biblio returns OPAC-filtered MARCXML: docker exec ktdmaster-koha-1 curl -sS -G 'http://127.0.0.1:2100/biblios' \ --data-urlencode 'version=1.1' \ --data-urlencode 'operation=searchRetrieve' \ --data-urlencode 'query=rec.id=1' \ --data-urlencode 'recordSchema=marcxml' \ --data-urlencode 'recordPacking=xml' \ --data-urlencode 'maximumRecords=1' 6. Confirm that a direct request for the suppressed biblio returns no hit and does not disclose suppression: docker exec ktdmaster-koha-1 curl -sS -G 'http://127.0.0.1:2100/biblios' \ --data-urlencode 'version=1.1' \ --data-urlencode 'operation=searchRetrieve' \ --data-urlencode 'query=rec.id=191' \ --data-urlencode 'recordSchema=marcxml' \ --data-urlencode 'recordPacking=xml' \ --data-urlencode 'maximumRecords=1' 7. Confirm that a normal search cannot disclose the suppressed biblio: docker exec ktdmaster-koha-1 curl -sS -G 'http://127.0.0.1:2100/biblios' \ --data-urlencode 'version=1.1' \ --data-urlencode 'operation=searchRetrieve' \ --data-urlencode 'query=rec.id=1 or rec.id=191' \ --data-urlencode 'recordSchema=marcxml' \ --data-urlencode 'recordPacking=xml' \ --data-urlencode 'maximumRecords=10' 8. Enable the blocked-record OpacSuppressionRedirect setting and repeat the direct request. It must contain Record is suppressed. The normal search must still hide biblio 191. 9. Set OpacSuppressionByIPRange to 127\\.0\\.0\\. and repeat the direct request from the container. It must return biblio 191. Disable OpacSuppression and confirm that the biblio also remains visible. 10. Set PublicSRUXSLT to /kohadevbox/koha/koha-tmpl/intranet-tmpl/prog/en/xslt/MARC21slim245a.xsl, restart the responder, and confirm that the visible biblio has only 245$a in its MARCXML. Verify an invalid stylesheet returns the temporary MARCXML transformation diagnostic. 11. Configure an OpacHiddenItems rule and a matching item. Confirm that it is not exposed, including from an IP-range exception. 12. Confirm the shared responder behaviour over Z39.50: printf '%s\n' 'base biblios' 'find @attr 1=12 191' 'show 1' 'quit' | \ docker exec -i ktdmaster-koha-1 yaz-client 127.0.0.1:2100 13. Run: prove -v t/Koha/Z3950Responder/PublicSRU.t prove -v t/db_dependent/Koha/Z3950Responder/GenericSession.t Co-authored-by: OpenAI Codex <codex@openai.com> -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 Mark Hofstetter <mark@hofstetter.at> changed: What |Removed |Added ---------------------------------------------------------------------------- Sponsorship status|--- |Sponsored Comma delimited| |Steiermärkische list of Sponsors| |Landesbibliothek -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 Mark Hofstetter <mark@hofstetter.at> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #206025|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 --- Comment #10 from Mark Hofstetter <mark@hofstetter.at> --- Created attachment 206026 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=206026&action=edit Bug 42952: Add public SRU configuration Add opt-in OPAC visibility rules for bibliographic requests received by the shared Z39.50/SRU responder when Elasticsearch is used. Suppressed and OPAC-hidden records are not exposed. A direct numeric rec.id request returns a suppression diagnostic only when OpacSuppressionRedirect uses the blocked-record response; otherwise it returns no hit. Add PublicSRUXSLT for optional MARCXML transformations after OPAC MARC filtering. Include a demo stylesheet that returns only 245$a, and return a temporary diagnostic when a stylesheet fails or does not return MARCXML. Document the configuration and add tests for all preference, suppression, redirect, IP-range, OPAC-hidden-item, XSLT, and Elasticsearch cases. Test plan: 1. Apply the patch, run database updates, and use Elasticsearch. 2. Restart the responder after changing preferences: koha-z3950-responder --restart kohadev 3. Use visible biblio 1 and set biblio 191 to opac_suppressed. 4. Set PublicSRU to Use, OpacSuppression to Hide, clear OpacSuppressionByIPRange, set OpacSuppressionRedirect to the not-found setting, and leave PublicSRUXSLT empty. 5. Confirm that a visible biblio returns OPAC-filtered MARCXML: docker exec ktdmaster-koha-1 curl -sS -G 'http://127.0.0.1:2100/biblios' \ --data-urlencode 'version=1.1' \ --data-urlencode 'operation=searchRetrieve' \ --data-urlencode 'query=rec.id=1' \ --data-urlencode 'recordSchema=marcxml' \ --data-urlencode 'recordPacking=xml' \ --data-urlencode 'maximumRecords=1' 6. Confirm that a direct request for the suppressed biblio returns no hit and does not disclose suppression: docker exec ktdmaster-koha-1 curl -sS -G 'http://127.0.0.1:2100/biblios' \ --data-urlencode 'version=1.1' \ --data-urlencode 'operation=searchRetrieve' \ --data-urlencode 'query=rec.id=191' \ --data-urlencode 'recordSchema=marcxml' \ --data-urlencode 'recordPacking=xml' \ --data-urlencode 'maximumRecords=1' 7. Confirm that a normal search cannot disclose the suppressed biblio: docker exec ktdmaster-koha-1 curl -sS -G 'http://127.0.0.1:2100/biblios' \ --data-urlencode 'version=1.1' \ --data-urlencode 'operation=searchRetrieve' \ --data-urlencode 'query=rec.id=1 or rec.id=191' \ --data-urlencode 'recordSchema=marcxml' \ --data-urlencode 'recordPacking=xml' \ --data-urlencode 'maximumRecords=10' 8. Enable the blocked-record OpacSuppressionRedirect setting and repeat the direct request. It must contain Record is suppressed. The normal search must still hide biblio 191. 9. Set OpacSuppressionByIPRange to 127\\.0\\.0\\. and repeat the direct request from the container. It must return biblio 191. Disable OpacSuppression and confirm that the biblio also remains visible. 10. Set PublicSRUXSLT to /kohadevbox/koha/koha-tmpl/intranet-tmpl/prog/en/xslt/MARC21slim245a.xsl, restart the responder, and confirm that the visible biblio has only 245$a in its MARCXML. Verify an invalid stylesheet returns the temporary MARCXML transformation diagnostic. 11. Configure an OpacHiddenItems rule and a matching item. Confirm that it is not exposed, including from an IP-range exception. 12. Confirm the shared responder behaviour over Z39.50: printf '%s\n' 'base biblios' 'find @attr 1=12 191' 'show 1' 'quit' | \ docker exec -i ktdmaster-koha-1 yaz-client 127.0.0.1:2100 13. Run: prove -v t/Koha/Z3950Responder/PublicSRU.t prove -v t/db_dependent/Koha/Z3950Responder/GenericSession.t Co-authored-by: OpenAI Codex <codex@openai.com> Sponsored-by: Steiermärkische Landesbibliothek -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 --- Comment #11 from Mark Hofstetter <mark@hofstetter.at> --- I didnt add Authentication, because "this should be done properly" (tm) using koha users + permissions which would put a little bit too much into one issue. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 --- Comment #12 from David Cook <dcook@prosentient.com.au> --- (In reply to Mark Hofstetter from comment #11)
I didnt add Authentication, because "this should be done properly" (tm) using koha users + permissions which would put a little bit too much into one issue.
Yeah, I also don't think we really need it for public SRU anyway at this stage. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |In Discussion Assignee|koha-bugs@lists.koha-commun |mark@hofstetter.at |ity.org | --- Comment #13 from David Cook <dcook@prosentient.com.au> --- Just moving this to "In Discussion" since you haven't marked it as Needs Signoff. Did you mean to put it into "Needs Signoff"? If so, I'd mark it as Failed QA because of some security issues. I'll add you to those, so you can see what changes will need to be made for this. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 --- Comment #14 from David Cook <dcook@prosentient.com.au> --- Really excited to see this work though. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 --- Comment #15 from David Cook <dcook@prosentient.com.au> --- Ah, also... did you add the XSLT so that they could control which fields are included? I think what you really want here is the Koha::RecordProcessor, as that's what applies the MARC Bibliographic Framework visibility rules. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 --- Comment #16 from David Cook <dcook@prosentient.com.au> --- I've had a bit of fun locally working with z3950_responder.pl... In /etc/koha/sites/kohadev/z3950/config.xml, I've added the following listener: <listen id="public">unix:/var/run/koha/kohadev/z3950_responder.sock</listen> And then in /etc/apache2/sites-enabled/kohadev.conf I've added the following before the Includes for the OAPC: ProxyPass /sru "unix:/var/run/koha/${instance}/z3950_responder.sock|http://localhost:8080" After you restart your z3950_responder.pl and your Apache, you can then do queries like this: http://localhost:8080/sru/biblios?operation=searchRetrieve&version=1.1&query=title=e&recordSchema=marcxml&startRecord=1&maximumRecords=20 -- Comparing the Elasticsearch vs the Zebra backends for Koha::Z3950Responder... there might already be some honouring of OPAC business logic... this needs review... -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 --- Comment #17 from David Cook <dcook@prosentient.com.au> --- (In reply to David Cook from comment #16)
ProxyPass /sru "unix:/var/run/koha/${instance}/z3950_responder.sock|http://localhost:8080"
Very important note...it is ESSENTIAL that your URL in the ProxyPass contains the port number for your VirtualHost. If it doesn't, for some inexplicable reason Apache silently will send all of your OPAC HTTP traffic to /var/run/koha/${instance}/z3950_responder.sock . Guess how I know and how many hairs I pulled out before figuring that out... -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 --- Comment #18 from David Cook <dcook@prosentient.com.au> --- (In reply to David Cook from comment #16)
Comparing the Elasticsearch vs the Zebra backends for Koha::Z3950Responder... there might already be some honouring of OPAC business logic... this needs review...
Nope... when using Elasticsearch and z3950_responder.pl plus OpacSuppression, it still shows the record... and that's because build_query_compat() takes a suppress argument. Also an is_opac argument... although that's just for OPAC specific indexes for ES... OpacHiddenItems in Koha/SearchEngine/Elasticsearch/Search.pm is based off C4::Context->interface being 'opac' which is problematic in this context... And then we don't apply MARC Bibliographic Frameworks via the ViewPolicy RecordProcessor filter because that's applied by C4::Search::searchResults() which does many many things that we wouldn't need... -- I suppose the difficulty with z3950_responder.pl is how do we instruct it to apply this "OPAC" logic to some requests but not others... I suppose we could pass an --opac flag with the koha-z3950-responder command... or something like that... some thought to go into there. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 --- Comment #19 from David Cook <dcook@prosentient.com.au> --- Or if you want to limit by IP address <Location /sru> <RequireAny> Require ip 192.168.1.100 Require all denied </RequireAny> ProxyPass "unix:/var/run/koha/${instance}/z3950_responder.sock|http://localhost:8080" </Location> Note that you might need to use mod_remoteip to get the real client IP depending on your setup. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 --- Comment #20 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Just curious how the z3950_responder does in terms of performance here (having the older fast reply of Zebra in mind)? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 --- Comment #21 from David Cook <dcook@prosentient.com.au> --- (In reply to Marcel de Rooy from comment #20)
Just curious how the z3950_responder does in terms of performance here (having the older fast reply of Zebra in mind)?
It's a good question. Logically, I think that it would have to be slower. However, I doubt it would be significantly worse. When I query the SRU endpoint from MarcEdit, using z3950_responder.pl with a Unix socket and Elasticsearch, it's still very fast. The more post-processing we add, the slower it'll get, but we can take that into account as we apply those layers I think. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42952 --- Comment #22 from David Cook <dcook@prosentient.com.au> --- (In reply to David Cook from comment #21)
(In reply to Marcel de Rooy from comment #20)
Just curious how the z3950_responder does in terms of performance here (having the older fast reply of Zebra in mind)?
It's a good question. Logically, I think that it would have to be slower. However, I doubt it would be significantly worse.
When I query the SRU endpoint from MarcEdit, using z3950_responder.pl with a Unix socket and Elasticsearch, it's still very fast.
The more post-processing we add, the slower it'll get, but we can take that into account as we apply those layers I think.
The MarcEdit use case is an interesting one though. In that context, we want to get the same MARCXML that we'd get in the cataloguing editor so that we don't lose any data. But in a public context we'd want the filtered MARCXML. I'm onboarding a hosted Koha library at the moment where they've used MarcEdit for many years with in-house Koha to do their cataloguing. I actually really like it as a workflow, and I'm hoping that we can use Marva Quartz in the same way to provide BIBFRAME cataloguing for Koha... (I just need to find the time to work on that instead of the million other Koha things I need to do...) -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org