[Koha-devel] Improving the API for record imports (Tomas Cohen Arazi)
David Cook
dcook at prosentient.com.au
Wed Sep 28 04:07:57 CEST 2016
That's cool. You should open a bug report!
I was thinking of doing something more sophisticated though. In my case, I
want to ingest an OAI-PMH record, transform it into MARCXML, match it
against any existing bibliographic records, and then do the add/update.
I also wonder if it could be more extensible as well... to ingest any type
of metadata record, transform it into MARCXML (or any internal metadata
format that Koha might use), match it against any existing internal records,
and then do an add/update.
Since the transformation process can be difficult, it would probably be wise
to store the original metadata record. I suppose it wouldn’t have to be
though. If the transformation failed, then your API call fails. With my
current work, I let the API call succeed, while providing a response that
says there was a problem. The problem then gets mitigated via a user
interface in Koha.
Maybe the thing to do is provider another route for a matcher, and have the
biblio CRUD route. It would mean 2x the HTTP requests if you need to match,
so that would slow things down... but it would separate out the logic more.
David Cook
Systems Librarian
Prosentient Systems
72/330 Wattle St
Ultimo, NSW 2007
Australia
Office: 02 9212 0899
Direct: 02 8005 0595
> -----Original Message-----
> From: koha-devel-bounces at lists.koha-community.org [mailto:koha-devel-
> bounces at lists.koha-community.org] On Behalf Of Benjamin Rokseth
> Sent: Tuesday, 27 September 2016 8:10 PM
> To: koha-devel at lists.koha-community.org
> Subject: Re: [Koha-devel] Improving the API for record imports (Tomas
> Cohen Arazi)
>
> Hi, Tomas and David,
>
> we have already implemented and are using a biblio CRUD endpoint, and I
> know the Finland community has, too.
> Sorry for not starting with a koha bug, but we simply didn't have the
time...
> Now, we would very much like to add /change this to a Koha native bug, but
> for now, heres our local patch:
> https://github.com/digibib/koha-docker/blob/master/koha-
> patched/patches/130-API-biblios-CRUD-expanded.patch
> which is based on koha master.
>
> Mainly, it reproduces the svc/biblio endpoint using marcxml for
add/update,
> and some other expansions for GETting an expanded record with items. It
> was quite easy to implement using Koha Objects.
>
> Please don't hesitate to shoot us, I mean, ask us for more!
>
> Benjamin Rokseth
> Oslo Public Library
> ________________________________________
> Fra: koha-devel-bounces at lists.koha-community.org <koha-devel-
> bounces at lists.koha-community.org> på vegne av koha-devel-
> request at lists.koha-community.org <koha-devel-request at lists.koha-
> community.org>
> Sendt: 27. september 2016 12:00
> Til: koha-devel at lists.koha-community.org
> Emne: Koha-devel Digest, Vol 130, Issue 56
>
> Send Koha-devel mailing list submissions to
> koha-devel at lists.koha-community.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
> or, via email, send a message with subject or body 'help' to
> koha-devel-request at lists.koha-community.org
>
> You can reach the person managing the list at
> koha-devel-owner at lists.koha-community.org
>
> When replying, please edit your Subject line so it is more specific than
"Re:
> Contents of Koha-devel digest..."
>
>
> Today's Topics:
>
> 1. Re: Improving the API for record imports (Tomas Cohen Arazi)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Mon, 26 Sep 2016 12:56:20 +0000
> From: Tomas Cohen Arazi <tomascohen at gmail.com>
> To: David Cook <dcook at prosentient.com.au>, Koha Devel
> <koha-devel at lists.koha-community.org>
> Subject: Re: [Koha-devel] Improving the API for record imports
> Message-ID:
>
> <CABZfb=WGH0gCH5veYjHmQEc4n1t+eoLsUmtUwMRceqW3tksLsA at mail.g
> mail.com>
> Content-Type: text/plain; charset="utf-8"
>
> David, I agree with your ideas. Having a REST endpoint handling records
> would be awesome improvement. I'll hapilly sign or QA those patches.
>
> Regards
>
> El mi?., 21 sept. 2016 a las 22:00, David Cook
(<dcook at prosentient.com.au>)
> escribi?:
>
> > Hi all,
> >
> >
> >
> > I want to improve the API(s) for importing (bibliographic) records.
> >
> >
> >
> > But I?m not 100% sure how to go about what criteria to use to gauge
> > ?improvement?. Here are some ideas:
> >
> >
> >
> > ? I think the REST API offers up one obvious addition and that?s
> > DELETE. It would be great to delete records using the API. I?ve found
> > this mostly thanks to OAI-PMH where you get a notice when a record has
> > been deleted upstream and needs to be deleted from Koha too.
> >
> > o The difficulty in deleting bibliographic records is items. If there
> > are items, it blocks the delete. You could auto-delete items, but you
> > can?t if there are holds or if it?s currently issued. At best, you
> > could return a failed delete via the API, I think.
> >
> >
> >
> > ? I keep thinking that ?import profiles? could be a good way to
> > go too. This might include a MARC Modification Template, it might be a
> > XSLT, it might be a Koha::Filter. It would let you turns your incoming
> > metadata from format A to format B.
> >
> > o This would also allow you to POST a minimum amount of data to the
> > API. You?d have your metadata record and your import profile code.
> >
> > ? The downside of this is the POST wouldn?t detail the options for
> > the import, and those imports would be more obfuscated. Changes to a
> > profile could also impact subsequent imports unintentionally.
> >
> > o I?m not sure how batches would work with filtering. You might have
> > 50/100 records filter correctly and import, but that leaves 50 which
> > aren?t imported. Do you undo the whole import, or do you fix your
> > filter, and then re-import the first 50 alongside the first time import
of the
> last 50?
> >
> >
> >
> > ? The original metadata would be stored regardless of metadata
> > format (e.g. marcxml, mods, rdf, dc, etc.) for historical purposes or
> > to allow re-filtering in the event that a filter fails to correctly
> > transform a record to valid MARCXML (or whatever internal metadata
> format Koha uses).
> >
> >
> >
> > ? The import source and the type of import would be stored (ie
> > http://oai-server.com and ?oai-pmh?, lx2.loc.gov and ?z3950?, <insert
> > relevant source> and ?oclc connexion?, etc.).
> >
> > o I think this would be helpful for managing/processing a certain type
> > of import, perhaps also in terms of matching records.
> >
> > o At the moment, with my work on #10662, I have a separate table
> > ?import_oai? which obviously is for records imported via oai-pmh only.
> > I don?t actually track the origin due to how I currently source the
> > imports, but I could do it in the future.
> >
> >
> >
> > ? Storing some sort of identifier would be good
> >
> > o OAI-PMH records have unique identifiers for the records
> >
> > o Z39.50? that?s tougher since it?s not a specific metadata format
like
> > the OAI-PMH format. So I don?t know about this one.
> >
> >
> >
> > Anyway, these are just some ideas I?ve been having. They?re not fully
> > formed by any stretch of the imagination.
> >
> >
> >
> > David Cook
> >
> > Systems Librarian
> >
> > Prosentient Systems
> >
> > 72/330 Wattle St
> >
> > Ultimo, NSW 2007
> >
> > Australia
> >
> >
> >
> > Office: 02 9212 0899
> >
> > Direct: 02 8005 0595
> >
> >
> > _______________________________________________
> > Koha-devel mailing list
> > Koha-devel at lists.koha-community.org
> > http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
> > website : http://www.koha-community.org/ git :
> > http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
>
> --
> Tom?s Cohen Arazi
> Theke Solutions (https://theke.io <http://theke.io/>) ? +54 9351 3513384
> GPG: B2F3C15F
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://lists.koha-community.org/pipermail/koha-
> devel/attachments/20160926/850b63b1/attachment-0001.html>
>
> ------------------------------
>
> _______________________________________________
> Koha-devel mailing list
> Koha-devel at lists.koha-community.org
> http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
> website : http://www.koha-community.org/ git : http://git.koha-
> community.org/ bugs : http://bugs.koha-community.org/
>
> End of Koha-devel Digest, Vol 130, Issue 56
> *******************************************
> _______________________________________________
> Koha-devel mailing list
> Koha-devel at lists.koha-community.org
> http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
> website : http://www.koha-community.org/ git : http://git.koha-
> community.org/ bugs : http://bugs.koha-community.org/
More information about the Koha-devel
mailing list