Re: [Koha-devel] Improving the API for record imports (Tomas Cohen Arazi)
Hi, Tomas and David, we have already implemented and are using a biblio CRUD endpoint, and I know the Finland community has, too. Sorry for not starting with a koha bug, but we simply didn't have the time... Now, we would very much like to add /change this to a Koha native bug, but for now, heres our local patch: https://github.com/digibib/koha-docker/blob/master/koha-patched/patches/130-... which is based on koha master. Mainly, it reproduces the svc/biblio endpoint using marcxml for add/update, and some other expansions for GETting an expanded record with items. It was quite easy to implement using Koha Objects. Please don't hesitate to shoot us, I mean, ask us for more! Benjamin Rokseth Oslo Public Library ________________________________________ Fra: koha-devel-bounces@lists.koha-community.org <koha-devel-bounces@lists.koha-community.org> på vegne av koha-devel-request@lists.koha-community.org <koha-devel-request@lists.koha-community.org> Sendt: 27. september 2016 12:00 Til: koha-devel@lists.koha-community.org Emne: Koha-devel Digest, Vol 130, Issue 56 Send Koha-devel mailing list submissions to koha-devel@lists.koha-community.org To subscribe or unsubscribe via the World Wide Web, visit http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel or, via email, send a message with subject or body 'help' to koha-devel-request@lists.koha-community.org You can reach the person managing the list at koha-devel-owner@lists.koha-community.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Koha-devel digest..." Today's Topics: 1. Re: Improving the API for record imports (Tomas Cohen Arazi) ---------------------------------------------------------------------- Message: 1 Date: Mon, 26 Sep 2016 12:56:20 +0000 From: Tomas Cohen Arazi <tomascohen@gmail.com> To: David Cook <dcook@prosentient.com.au>, Koha Devel <koha-devel@lists.koha-community.org> Subject: Re: [Koha-devel] Improving the API for record imports Message-ID: <CABZfb=WGH0gCH5veYjHmQEc4n1t+eoLsUmtUwMRceqW3tksLsA@mail.gmail.com> Content-Type: text/plain; charset="utf-8" David, I agree with your ideas. Having a REST endpoint handling records would be awesome improvement. I'll hapilly sign or QA those patches. Regards El mi?., 21 sept. 2016 a las 22:00, David Cook (<dcook@prosentient.com.au>) escribi?:
Hi all,
I want to improve the API(s) for importing (bibliographic) records.
But I?m not 100% sure how to go about what criteria to use to gauge ?improvement?. Here are some ideas:
? I think the REST API offers up one obvious addition and that?s DELETE. It would be great to delete records using the API. I?ve found this mostly thanks to OAI-PMH where you get a notice when a record has been deleted upstream and needs to be deleted from Koha too.
o The difficulty in deleting bibliographic records is items. If there are items, it blocks the delete. You could auto-delete items, but you can?t if there are holds or if it?s currently issued. At best, you could return a failed delete via the API, I think.
? I keep thinking that ?import profiles? could be a good way to go too. This might include a MARC Modification Template, it might be a XSLT, it might be a Koha::Filter. It would let you turns your incoming metadata from format A to format B.
o This would also allow you to POST a minimum amount of data to the API. You?d have your metadata record and your import profile code.
? The downside of this is the POST wouldn?t detail the options for the import, and those imports would be more obfuscated. Changes to a profile could also impact subsequent imports unintentionally.
o I?m not sure how batches would work with filtering. You might have 50/100 records filter correctly and import, but that leaves 50 which aren?t imported. Do you undo the whole import, or do you fix your filter, and then re-import the first 50 alongside the first time import of the last 50?
? The original metadata would be stored regardless of metadata format (e.g. marcxml, mods, rdf, dc, etc.) for historical purposes or to allow re-filtering in the event that a filter fails to correctly transform a record to valid MARCXML (or whatever internal metadata format Koha uses).
? The import source and the type of import would be stored (ie http://oai-server.com and ?oai-pmh?, lx2.loc.gov and ?z3950?, <insert relevant source> and ?oclc connexion?, etc.).
o I think this would be helpful for managing/processing a certain type of import, perhaps also in terms of matching records.
o At the moment, with my work on #10662, I have a separate table ?import_oai? which obviously is for records imported via oai-pmh only. I don?t actually track the origin due to how I currently source the imports, but I could do it in the future.
? Storing some sort of identifier would be good
o OAI-PMH records have unique identifiers for the records
o Z39.50? that?s tougher since it?s not a specific metadata format like the OAI-PMH format. So I don?t know about this one.
Anyway, these are just some ideas I?ve been having. They?re not fully formed by any stretch of the imagination.
David Cook
Systems Librarian
Prosentient Systems
72/330 Wattle St
Ultimo, NSW 2007
Australia
Office: 02 9212 0899
Direct: 02 8005 0595
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Tom?s Cohen Arazi Theke Solutions (https://theke.io <http://theke.io/>) ? +54 9351 3513384 GPG: B2F3C15F
El mar., 27 sept. 2016 a las 7:10, Benjamin Rokseth (< benjamin.rokseth@kul.oslo.kommune.no>) escribió:
Hi, Tomas and David,
we have already implemented and are using a biblio CRUD endpoint, and I know the Finland community has, too. Sorry for not starting with a koha bug, but we simply didn't have the time... Now, we would very much like to add /change this to a Koha native bug, but for now, heres our local patch:
https://github.com/digibib/koha-docker/blob/master/koha-patched/patches/130-... which is based on koha master.
Ah, c'mon. File a bug report and attach your patch, we can work on top of it to make it fit our QA requirements. . -- Tomás Cohen Arazi Theke Solutions (https://theke.io <http://theke.io/>) ✆ +54 9351 3513384 GPG: B2F3C15F
That's cool. You should open a bug report! I was thinking of doing something more sophisticated though. In my case, I want to ingest an OAI-PMH record, transform it into MARCXML, match it against any existing bibliographic records, and then do the add/update. I also wonder if it could be more extensible as well... to ingest any type of metadata record, transform it into MARCXML (or any internal metadata format that Koha might use), match it against any existing internal records, and then do an add/update. Since the transformation process can be difficult, it would probably be wise to store the original metadata record. I suppose it wouldnt have to be though. If the transformation failed, then your API call fails. With my current work, I let the API call succeed, while providing a response that says there was a problem. The problem then gets mitigated via a user interface in Koha. Maybe the thing to do is provider another route for a matcher, and have the biblio CRUD route. It would mean 2x the HTTP requests if you need to match, so that would slow things down... but it would separate out the logic more. David Cook Systems Librarian Prosentient Systems 72/330 Wattle St Ultimo, NSW 2007 Australia Office: 02 9212 0899 Direct: 02 8005 0595
-----Original Message----- From: koha-devel-bounces@lists.koha-community.org [mailto:koha-devel- bounces@lists.koha-community.org] On Behalf Of Benjamin Rokseth Sent: Tuesday, 27 September 2016 8:10 PM To: koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Improving the API for record imports (Tomas Cohen Arazi)
Hi, Tomas and David,
we have already implemented and are using a biblio CRUD endpoint, and I know the Finland community has, too. Sorry for not starting with a koha bug, but we simply didn't have the time... Now, we would very much like to add /change this to a Koha native bug, but for now, heres our local patch: https://github.com/digibib/koha-docker/blob/master/koha- patched/patches/130-API-biblios-CRUD-expanded.patch which is based on koha master.
Mainly, it reproduces the svc/biblio endpoint using marcxml for add/update, and some other expansions for GETting an expanded record with items. It was quite easy to implement using Koha Objects.
Please don't hesitate to shoot us, I mean, ask us for more!
Benjamin Rokseth Oslo Public Library ________________________________________ Fra: koha-devel-bounces@lists.koha-community.org <koha-devel- bounces@lists.koha-community.org> på vegne av koha-devel- request@lists.koha-community.org <koha-devel-request@lists.koha- community.org> Sendt: 27. september 2016 12:00 Til: koha-devel@lists.koha-community.org Emne: Koha-devel Digest, Vol 130, Issue 56
Send Koha-devel mailing list submissions to koha-devel@lists.koha-community.org
To subscribe or unsubscribe via the World Wide Web, visit
or, via email, send a message with subject or body 'help' to koha-devel-request@lists.koha-community.org
You can reach the person managing the list at koha-devel-owner@lists.koha-community.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of Koha-devel digest..."
Today's Topics:
1. Re: Improving the API for record imports (Tomas Cohen Arazi)
----------------------------------------------------------------------
Message: 1 Date: Mon, 26 Sep 2016 12:56:20 +0000 From: Tomas Cohen Arazi <tomascohen@gmail.com> To: David Cook <dcook@prosentient.com.au>, Koha Devel <koha-devel@lists.koha-community.org> Subject: Re: [Koha-devel] Improving the API for record imports Message-ID:
<CABZfb=WGH0gCH5veYjHmQEc4n1t+eoLsUmtUwMRceqW3tksLsA@mail.g mail.com> Content-Type: text/plain; charset="utf-8"
David, I agree with your ideas. Having a REST endpoint handling records would be awesome improvement. I'll hapilly sign or QA those patches.
Regards
El mi?., 21 sept. 2016 a las 22:00, David Cook (<dcook@prosentient.com.au>) escribi?:
Hi all,
I want to improve the API(s) for importing (bibliographic) records.
But I?m not 100% sure how to go about what criteria to use to gauge ?improvement?. Here are some ideas:
? I think the REST API offers up one obvious addition and that?s DELETE. It would be great to delete records using the API. I?ve found this mostly thanks to OAI-PMH where you get a notice when a record has been deleted upstream and needs to be deleted from Koha too.
o The difficulty in deleting bibliographic records is items. If there are items, it blocks the delete. You could auto-delete items, but you can?t if there are holds or if it?s currently issued. At best, you could return a failed delete via the API, I think.
? I keep thinking that ?import profiles? could be a good way to go too. This might include a MARC Modification Template, it might be a XSLT, it might be a Koha::Filter. It would let you turns your incoming metadata from format A to format B.
o This would also allow you to POST a minimum amount of data to the API. You?d have your metadata record and your import profile code.
? The downside of this is the POST wouldn?t detail the options for the import, and those imports would be more obfuscated. Changes to a profile could also impact subsequent imports unintentionally.
o I?m not sure how batches would work with filtering. You might have 50/100 records filter correctly and import, but that leaves 50 which aren?t imported. Do you undo the whole import, or do you fix your filter, and then re-import the first 50 alongside the first time import of the last 50?
? The original metadata would be stored regardless of metadata format (e.g. marcxml, mods, rdf, dc, etc.) for historical purposes or to allow re-filtering in the event that a filter fails to correctly transform a record to valid MARCXML (or whatever internal metadata format Koha uses).
? The import source and the type of import would be stored (ie http://oai-server.com and ?oai-pmh?, lx2.loc.gov and ?z3950?, <insert relevant source> and ?oclc connexion?, etc.).
o I think this would be helpful for managing/processing a certain type of import, perhaps also in terms of matching records.
o At the moment, with my work on #10662, I have a separate table ?import_oai? which obviously is for records imported via oai-pmh only. I don?t actually track the origin due to how I currently source the imports, but I could do it in the future.
? Storing some sort of identifier would be good
o OAI-PMH records have unique identifiers for the records
o Z39.50? that?s tougher since it?s not a specific metadata format
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel like
the OAI-PMH format. So I don?t know about this one.
Anyway, these are just some ideas I?ve been having. They?re not fully formed by any stretch of the imagination.
David Cook
Systems Librarian
Prosentient Systems
72/330 Wattle St
Ultimo, NSW 2007
Australia
Office: 02 9212 0899
Direct: 02 8005 0595
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Tom?s Cohen Arazi Theke Solutions (https://theke.io <http://theke.io/>) ? +54 9351 3513384 GPG: B2F3C15F
participants (3)
-
Benjamin Rokseth -
David Cook -
Tomas Cohen Arazi