[Bug 40308] New: New Perl Class (Koha::ExternalContent::OCLC) for integrating with OCLC's APIs
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40308 Bug ID: 40308 Summary: New Perl Class (Koha::ExternalContent::OCLC) for integrating with OCLC's APIs Change sponsored?: --- Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: MARC Authority data support Assignee: koha-bugs@lists.koha-community.org Reporter: a.roussos@dataly.gr QA Contact: testopia@bugs.koha-community.org Background information ---------------------- OCLC recently [1] started offering a Linked Data (LD) [2] API [3] that allows looking up Dewey Decimal Classification (DDC) numbers [4] in order to obtain unique URIs/URLs and Descriptions for a particular DDC (or range). In addition to API searches, a smaller subset of the LD can also be browsed online in a hierarchical manner [5]. This is just one from a large set of APIs offered by OCLC [6]. Reasoning behind this Bug report -------------------------------- We believe that Koha users who include DDC numbers in their authority cataloguing would benefit from an integration with the OCLC Dewey LD API that would allow them to easily look up DDCs in exchange for LD URIs/URLs and Descriptions. To this end, we propose the addition of a new Perl Class that will offer this functionality (Koha::ExternalContent::OCLC). Initially, the class will offer functions for issuing API access tokens and for looking up DDC URIs/URLs and Descriptions on the Dewey LD API. In the future, more code be added to the class to allow for integration with more OCLC APIs. This Bug report will also include a command-line tool that will enable users to update their authorities with OCLC Linked Data en masse, useful for authority records that already contain DDC information. Three new authority Framework plugins that will use the functions offered by the new Perl Class will be added as separate Bug reports. These offer the ability to look up DDCs with a single mouse click while inside the Authorities module cataloguing editor. Finally, we propose the addition of four new System Preferences that control the integration with the OCLC Dewey Linked Data API, namely: - OCLCAPIWSKeyClientID: Stores the client ID component of your OCLC API WSKey - OCLCAPIWSKeyClientSecret: Stores the client secret component of your OCLC API WSKey - OCLCDeweyLinkedDataAPI: Basically an On/Off "switch" for the lookup service - OCLCDeweyLinkedDataLanguage: Sets the language of the DDC description to be fetched References: [1] https://www.oclc.org/en/news/releases/2024/20240815-uniform-resource-identif... [2] https://www.oclc.org/en/linked-data.html [3] https://developer.api.oclc.org/dewey-ld [4] https://www.oclc.org/content/dam/oclc/dewey/versions/print/intro.pdf [5] https://entities.oclc.org/worldcat/ddc/ [6] https://www.oclc.org/developer/api/oclc-apis.en.html -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40308 Andreas Roussos <a.roussos@dataly.gr> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED Assignee|koha-bugs@lists.koha-commun |a.roussos@dataly.gr |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40308 Andreas Roussos <a.roussos@dataly.gr> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |40309 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40309 [Bug 40309] New authority framework plugin for MARC21 (marc21_field_082a_authorities.pl) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40308 Andreas Roussos <a.roussos@dataly.gr> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |40310 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40310 [Bug 40310] New authority framework plugin for MARC21 (marc21_field_083a_authorities.pl) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40308 Andreas Roussos <a.roussos@dataly.gr> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |40311 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40311 [Bug 40311] New authority framework plugin for UNIMARC (unimarc_field_676a_authorities.pl) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40308 --- Comment #1 from Andreas Roussos <a.roussos@dataly.gr> --- Created attachment 183809 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=183809&action=edit Sample authority records (MARC21 flavour) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40308 --- Comment #2 from Andreas Roussos <a.roussos@dataly.gr> --- Created attachment 183810 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=183810&action=edit Sample authority records (UNIMARC flavour) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40308 Andreas Roussos <a.roussos@dataly.gr> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|--- |Medium patch Status|ASSIGNED |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40308 --- Comment #3 from Andreas Roussos <a.roussos@dataly.gr> --- Created attachment 183811 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=183811&action=edit Bug 40308: add new Perl Class Koha::ExternalContent::OCLC OCLC are offering a number of web APIs, this is an attempt to collect utility functions in a new Perl Class. At the moment, the Class contains the following functions, that allow interaction with the OCLC Dewey Linked Data API: - get_access_token - get_ddc_uri - get_ddc_url - get_ddc_description Test plan: 1) Inside a KTD instance (MARC flavour does not play a role), run `perldoc -o html Koha::ExternalContent::OCLC > OCLC.html` and view the documentation for this class (OCLC.html) in your web browser. Ensure everything makes sense ;-) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40308 --- Comment #4 from Andreas Roussos <a.roussos@dataly.gr> --- Created attachment 183812 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=183812&action=edit Bug 40308: add new System Preferences for OCLC Dewey API This patch adds four new System Preferences that control the integration with the OCLC Dewey Linked Data API, namely: - OCLCAPIWSKeyClientID: Stores the client ID component of your OCLC API WSKey - OCLCAPIWSKeyClientSecret: Stores the client secret component of your OCLC API WSKey - OCLCDeweyLinkedDataAPI: Basically an On/Off "switch" for the lookup service - OCLCDeweyLinkedDataLanguage: Sets the language of the DDC description to be fetched Test plan: 0) Launch either a MARC21 or a UNIMARC KTD container (MARC flavour does not play a role for this particular test plan), then apply this patch. 1) Run `reset_all` to force your KTD instance to load installer/data/mysql/mandatory/sysprefs.sql. 2) Then, visit Administration > System preferences > Web services and inspect the four new sysprefs under 'OCLC Dewey Linked Data API'. Look for typos in the descriptions, etc. 3) Then, run `updatedatabase` to test the atomic update. It should complete without warnings/errors. 4) Finally, visually inspect the output of `git log -p C4/UsageStats.pm` for errors. Notice that the System Preferences pertaining to the WSKey client ID and secret were *not* added for security reasons. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40308 --- Comment #5 from Andreas Roussos <a.roussos@dataly.gr> --- Created attachment 183813 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=183813&action=edit Bug 40308: add a command-line tool that updates authorities en masse In Bugs 40309, 40310, and 40311 we will add three new Authority Framework plugins that allow for easy DDC lookups using OCLC's Dewey Linked Data API while cataloguing authorities in the Staff Interface. However, users may want to retroactively enrich their existing authorities using a command-line tool that can modify authorities en masse. This patch adds misc/maintenance/fetch_oclc_dewey_linked_data.pl, which is a Perl script that can accept a number of options in order to facilitate selective operation on specific authority types, or even authid ranges. Test plan (NOTE: you need to have an organizational account with OCLC and an active WebDewey subscription associated with it in order to request a WSKey for accessing the Dewey Linked Data API): [ I recommend working with two KTD instances in parallel. ] 0) First of all, run: ktd_proxy --start 1) Then, edit your KTD .env file so that: KOHA_MARC_FLAVOUR=marc21 Then: ktd --proxy --name marc21 up Wait until it starts up. Then, in a different terminal, edit your KTD .env file so that: KOHA_MARC_FLAVOUR=unimarc Then: ktd --proxy --name unimarc up Wait until it starts up. You now have two KTDs running, one for each MARC flavour you want to test. 2) Get inside the MARC21 Koha container: ktd --name marc21 --shell Download the sample MARC21 authority records (sample_auth_records_marc21.mrc) attached to this Bug report (the file contains 15 records): wget -O sample_auth_records_marc21.mrc https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=183809 Then, import the authorities: misc/migration_tools/bulkmarcimport.pl -a --file sample_auth_records_marc21.mrc -v -c MARC21 -m ISO2709 Repeat for the UNIMARC Koha container: ktd --name unimarc --shell Download the sample UNIMARC authority records (sample_auth_records_unimarc.mrc) attached to this Bug report (the file contains 12 records): wget -O sample_auth_records_unimarc.mrc https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=183810 Then, import the authorities: misc/migration_tools/bulkmarcimport.pl -a --file sample_auth_records_unimarc.mrc -v -c UNIMARC -m ISO2709 3) Once the sample authorities have been imported, you must fill in these two System Preferences (this is the tricky part, as, in order to request a WSKey you need to have an organizational account with OCLC and an active WebDewey subscription associated with it): - OCLCAPIWSKeyClientID - OCLCAPIWSKeyClientSecret 4) When the above System Preferences have been filled in, try running the new script (passing as many variations of the command-line options as you can). For example: cd misc/maintenance/ ./fetch_oclc_dewey_linked_data.pl -h ./fetch_oclc_dewey_linked_data.pl --man | cat ./fetch_oclc_dewey_linked_data.pl -v ./fetch_oclc_dewey_linked_data.pl -v -l el ./fetch_oclc_dewey_linked_data.pl -v -a SNC ./fetch_oclc_dewey_linked_data.pl -v -a SNC -w "authid = 5955" ./fetch_oclc_dewey_linked_data.pl -v -a SNC -w "authid = 5955" -c ./fetch_oclc_dewey_linked_data.pl -v -a SNC -w "authid = 5955" -l it ./fetch_oclc_dewey_linked_data.pl -v -a SNC -w "authid = 5955" -l it -f ./fetch_oclc_dewey_linked_data.pl -v -l no -f ./fetch_oclc_dewey_linked_data.pl ...etc The output of the script for the above commands should resemble what is being depicted in the following asciinema screencast: https://asciinema.org/a/FKxWqNHLeZbxcgCBwYWZ4fod9 If all goes well, you should have updated all authorities that have a DDC field with their matching Dewey Linked Data. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40308 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40308 --- Comment #6 from Chris Cormack <chris@bigballofwax.co.nz> --- Automated patch testing results: [PASS] ALL PATCHES APPLY CLEANLY All patches apply successfully to the current codebase. **Patch Results:** [PASS] Bug-40308-add-a-command-line-tool-that-updates-aut.patch (ID: 183813) Status: SUCCESS + Patch applies cleanly [PASS] Bug-40308-add-new-System-Preferences-for-OCLC-Dewe.patch (ID: 183812) Status: SUCCESS + Patch applies cleanly [PASS] Bug-40308-add-new-Perl-Class-KohaExternalContentOC.patch (ID: 183811) Status: SUCCESS + Patch applies cleanly --- This is an automated message from the patch testing system. Tested against branch: main -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40308 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david@davidnind.com Status|Needs Signoff |Patch doesn't apply --- Comment #7 from David Nind <david@davidnind.com> --- The patch no longer applies 8-(... git bz apply 40308 Bug 40308 - New Perl Class (Koha::ExternalContent::OCLC) for integrating with OCLC's APIs 183811 - Bug 40308: add new Perl Class Koha::ExternalContent::OCLC 183812 - Bug 40308: add new System Preferences for OCLC Dewey API 183813 - Bug 40308: add a command-line tool that updates authorities en masse Apply? [(y)es, (n)o, (i)nteractive] y Applying: Bug 40308: add new Perl Class Koha::ExternalContent::OCLC Applying: Bug 40308: add new System Preferences for OCLC Dewey API Using index info to reconstruct a base tree... M C4/UsageStats.pm M installer/data/mysql/mandatory/sysprefs.sql Falling back to patching base and 3-way merge... Auto-merging installer/data/mysql/mandatory/sysprefs.sql CONFLICT (content): Merge conflict in installer/data/mysql/mandatory/sysprefs.sql Auto-merging C4/UsageStats.pm error: Failed to merge in the changes. Patch failed at 0001 Bug 40308: add new System Preferences for OCLC Dewey API As the bugs say, "you need to have an organizational account with OCLC and an active WebDewey subscription associated with it in order to request a WSKey for accessing the Dewey Linked Data API". So I'm not sure how to get this through sign-off, QA and the release manager without it. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40308 Andreas Roussos <a.roussos@dataly.gr> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #183812|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40308 --- Comment #8 from Andreas Roussos <a.roussos@dataly.gr> --- Created attachment 198240 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=198240&action=edit Bug 40308: add new System Preferences for OCLC Dewey API This patch adds four new System Preferences that control the integration with the OCLC Dewey Linked Data API, namely: - OCLCAPIWSKeyClientID: Stores the client ID component of your OCLC API WSKey - OCLCAPIWSKeyClientSecret: Stores the client secret component of your OCLC API WSKey - OCLCDeweyLinkedDataAPI: Basically an On/Off "switch" for the lookup service - OCLCDeweyLinkedDataLanguage: Sets the language of the DDC description to be fetched Test plan: 0) Launch either a MARC21 or a UNIMARC KTD container (MARC flavour does not play a role for this particular test plan), then apply this patch. 1) Run `reset_all` to force your KTD instance to load installer/data/mysql/mandatory/sysprefs.sql. 2) Then, visit Administration > System preferences > Web services and inspect the four new sysprefs under 'OCLC Dewey Linked Data API'. Look for typos in the descriptions, etc. 3) Then, run `updatedatabase` to test the atomic update. It should complete without warnings/errors. 4) Finally, visually inspect the output of `git log -p C4/UsageStats.pm` for errors. Notice that the System Preferences pertaining to the WSKey client ID and secret were *not* added for security reasons. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40308 --- Comment #9 from Andreas Roussos <a.roussos@dataly.gr> --- (In reply to David Nind from comment #7)
The patch no longer applies 8-(... I have rebased the failing patch on current main. It wasn't applying cleanly due to changes introduced by Bug 41834.
As the bugs say, "you need to have an organizational account with OCLC and an active WebDewey subscription associated with it in order to request a WSKey for accessing the Dewey Linked Data API".
So I'm not sure how to get this through sign-off, QA and the release manager without it. You're absolutely right. When I initially submitted this enhancement I was hoping there would be someone with access to such a key that could manage to test the proposed code changes.
I'll attend a meeting with OCLC Staff tomorrow where this is going to be on the agenda: we're going to discuss if/how it could be possible for a testing WSKey to be issued specifically for the purpose of testing this Bug and its dependencies. I'll keep you posted of any developments. Thanks! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40308 Andreas Roussos <a.roussos@dataly.gr> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |BLOCKED --- Comment #10 from Andreas Roussos <a.roussos@dataly.gr> --- Temporarily setting the Status to BLOCKED, until we can find a way of providing testers with a Client ID and Client Secret key pair so that they can access the OCLC Dewey Linked Data API. -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org