[Bug 25905] New: REST API: create endpoint for importing OAI-PMH records from external OAI-PMH clients
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25905 Bug ID: 25905 Summary: REST API: create endpoint for importing OAI-PMH records from external OAI-PMH clients Change sponsored?: --- Product: Koha Version: unspecified Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: REST API Assignee: koha-bugs@lists.koha-community.org Reporter: dcook@prosentient.com.au Following on from previous work on #10662 and koha-plugin-oaipmh-import, I'm now thinking that it would be best to have a REST API endpoint for importing OAI-PMH records in the Koha mainstream. This would give libraries assurance that the core OAI-PMH ingest functionality will be in the main codebase and get Koha community support. For now, I'm still planning to keep my planned OAI-PMH harvester/client as a plugin/third-party dependency, but completion of this work will allow Koha libraries to use any OAI-PMH harvester they choose to ingest records into Koha. This could be with a Perl cronjob using HTTP::OAI, or it could be a Golang daemon, or whatever. Lots of possibilities. Anyway, I've got a lot of this work done already in koha-plugin-oaipmh-import, so I'll look at re-organising it to fit into Koha proper. It'll be in my own time, so I don't have a timeline for it, but it is on my TODO list. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25905 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=10662 -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25905 Andreas Hedström Mace <andreas.hedstrom.mace@sub.su.se> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |andreas.hedstrom.mace@sub.s | |u.se --- Comment #1 from Andreas Hedström Mace <andreas.hedstrom.mace@sub.su.se> --- I think this would be really good to have. A stable endpoint to ingest the records, and the harvester as a plug-in (or even various harvester plug-ins). -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25905 --- Comment #2 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Hi David, can you please write up an RFC for the new endpoint to be voted on? https://wiki.koha-community.org/wiki/REST_api_RFCs We are trying to make the new API as consistent and clean as possible. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25905 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |tomascohen@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25905 --- Comment #3 from David Cook <dcook@prosentient.com.au> --- (In reply to Katrin Fischer from comment #2)
Hi David, can you please write up an RFC for the new endpoint to be voted on? https://wiki.koha-community.org/wiki/REST_api_RFCs We are trying to make the new API as consistent and clean as possible.
Absolutely. That sounds great to me. Once I've written up the RFC, do I need to email someone or is someone tracking changes on that page? -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25905 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |dcook@prosentient.com.au |ity.org | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25905 --- Comment #4 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- You can email and put it on the next dev meetings agenda for discussion/vote. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25905 --- Comment #5 from David Cook <dcook@prosentient.com.au> --- (In reply to Katrin Fischer from comment #4)
You can email and put it on the next dev meetings agenda for discussion/vote.
Great. I don't have a timeline for this at the moment, but I'll keep it in mind. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25905 --- Comment #6 from David Cook <dcook@prosentient.com.au> --- Ok I've done a little RFC. Referenced at: https://wiki.koha-community.org/wiki/REST_api_RFCs#Endpoints Found at: https://wiki.koha-community.org/wiki/Import_biblios_oaipmh_endpoint_RFC -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25905 --- Comment #7 from David Cook <dcook@prosentient.com.au> --- I never heard back here or on koha-devel about the API RFC, so I'm just going to keep going ahead with this one. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25905 --- Comment #8 from David Cook <dcook@prosentient.com.au> --- Annnd I just had another thought for this which could make it more robust. At the moment, I'm planning on doing a synchronous import. However, when we have RabbitMQ, it would be cool to have the API just stage the import, and let a background worker do the actual import work. In this scenario, a person could have the API endpoint actually return a transaction ID, which could be used to poll Koha for status on the import. That said, polling is better for human-machine interactions, rather than machine-machine interactions. And it wouldn't make sense for a harvester/downloader to wait for the transaction to complete anyway. Maybe I just shouldn't worry about it for now... -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25905 --- Comment #9 from David Cook <dcook@prosentient.com.au> --- Created attachment 107880 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=107880&action=edit Bug 25905: Add new 'import_oaipmh_biblios' table -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25905 --- Comment #10 from David Cook <dcook@prosentient.com.au> --- Created attachment 107881 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=107881&action=edit Bug 25905: DBIx::Class files -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25905 --- Comment #11 from David Cook <dcook@prosentient.com.au> --- Created attachment 107882 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=107882&action=edit Bug 25905: Create /import/oaipmh/biblios API endpoint -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25905 --- Comment #12 from David Cook <dcook@prosentient.com.au> --- Created attachment 107883 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=107883&action=edit Bug 25905: Tidy up warns -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25905 --- Comment #13 from David Cook <dcook@prosentient.com.au> --- These patches should work, but I've run out of time/energy tonight to write the unit tests. Test plan: 1. Turn on RESTBasicAuth 2. curl -XPOST -u username:password -H "x-koha-oaipmh-repository: http://koha-community.org/oai.pl" http://localhost:8081/api/v1/import/oaipmh/biblios -d @oaipmh.xml That oaipmh.xml file can be a ListRecords or GetRecord document. The API will handle add/update/delete. I'll look at providing some test files later too. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25905 --- Comment #14 from David Cook <dcook@prosentient.com.au> --- Once this is in place, I will keep working on my Golang-based harvester fronted by a Koha plugin. Once Koha is using RabbitMQ for a job queue, I'll probably change this API endpoint to stage the import, and then let the background worker complete the actual import. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25905 Michal Denar <black23@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |black23@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25905 Michal Denar <black23@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |josef.moravec@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25905 --- Comment #15 from David Cook <dcook@prosentient.com.au> --- (In reply to David Cook from comment #14)
Once Koha is using RabbitMQ for a job queue, I'll probably change this API endpoint to stage the import, and then let the background worker complete the actual import.
It looks like the future has caught up with me. Bug 22417 has been pushed to master, so I could make it so that the endpoint accepts and stages the records, and then passes messages to a background worker via RabbitMQ. It'll be more complicated but it will be much more robust and scalable. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25905 Lauren Denny <lauren_denny@sil.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lauren_denny@sil.org -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25905 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=35659 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25905 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution|--- |INVALID --- Comment #16 from David Cook <dcook@prosentient.com.au> --- With the availability of bug 35659, I don't think this bug report is necessary anymore. -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org