<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"><html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body>A new request with request id 5929 has been created by koha-devel-request@lists.koha-community.org. Short info on the request is : <br><br>Title : Koha-devel Digest, Vol 204, Issue 17<br>Category : <br>Description : <div>Send Koha-devel mailing list submissions to<br> koha-devel@lists.koha-community.org<br><br>To subscribe or unsubscribe via the World Wide Web, visit<br> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel<br>or, via email, send a message with subject or body 'help' to<br> koha-devel-request@lists.koha-community.org<br><br>You can reach the person managing the list at<br> koha-devel-owner@lists.koha-community.org<br><br>When replying, please edit your Subject line so it is more specific<br>than "Re: Contents of Koha-devel digest..."<br><br><br>Today's Topics:<br><br> 1. Re: [Koha] OAI-PMH harvester (Arthur Suzuki)<br><br><br>----------------------------------------------------------------------<br><br>Message: 1<br>Date: Wed, 23 Nov 2022 14:44:21 +0100<br>From: Arthur Suzuki <arthur.suzuki@biblibre.com><br>To: "Mike D." <black23@gmail.com>, BOUIS Sonia<br> <sonia.bouis@univ-lyon3.fr><br>Cc: koha@lists.katipo.co.nz, koha-devel@lists.koha-community.org<br>Subject: Re: [Koha-devel] [Koha] OAI-PMH harvester<br>Message-ID: <84e9b5bd9fb0e75f3fad18d0a8381961@biblibre.com><br>Content-Type: text/plain; charset=UTF-8; format=flowed<br><br>Hello there,<br>If I may suggest a good harvester library, Catmandu may do the job <br>pretty well.<br>I've not used the OAI module but used it to harvest from a JSON source <br>and transform to an UNIMARC file with pretty good success so far.<br>It can export seamlessly to iso2709 or marcxml.<br>https://metacpan.org/dist/Catmandu-OAI<br><br>Best,<br>Arthur<br><br>On 2022-11-22 15:57, Mike D. wrote:<br>> Hey. Hey,<br>> I'm really glad to see the OAI-PMH harvester debate going on for Koha. <br>> I<br>> think if we choose a good external harvester with support, we can save <br>> a<br>> lot of energy and resources to implement related activities in the <br>> system.<br>> Shoveling the logs is only part of the story. The easy part. Since the<br>> result of shoveling is a lot of records, most of the time we can't <br>> avoid<br>> post-processing, merging with the records in the local database. For<br>> example, if you need to update records from a source where there are<br>> millions of records, but there are hundreds of thousands in the local<br>> database. Only a slice of that huge amount is relevant. If we design <br>> the<br>> processing workflow wrong, it will take unnecessarily long and burn<br>> valuable resources.<br>> I would hereby like to invite us to be in touch, to debate and share <br>> our<br>> experiences. Let's get this area moving towards a successful finish.<br>> <br>> Take care.<br>> <br>> Michal<br>> <br>> út 22. 11. 2022 v 15:13 odesílatel BOUIS Sonia <br>> <sonia.bouis@univ-lyon3.fr><br>> napsal:<br>> <br>>> Hi,<br>>> Thanks to David, Tomas, Michal and Michael for your replies.<br>>> <br>>> So we have decided to evaluate several external OAI-PMH client that <br>>> could<br>>> be used by Koha and to choose one in the end of January<br>>> There a lot to do after that and we discussed about the background <br>>> jobs<br>>> and cronjobs seems to be appropriate. We thought that the settings in <br>>> the<br>>> koha intranet should be only to define URLs, SETs, or XSLT sheets (for<br>>> example, to transform DC XML in MARCXML).<br>>> <br>>> We are only at the begining of the process 😊<br>>> <br>>> Kind regards,<br>>> Sonia<br>>> <br>>> ------------------------------<br>>> <br>>> Message: 2<br>>> Date: Wed, 26 Oct 2022 10:37:49 +1100<br>>> From: "David Cook" <dcook@prosentient.com.au><br>>> To: "'Tomas Cohen Arazi'" <tomascohen@gmail.com>, "'BOUIS Sonia'"<br>>> <sonia.bouis@univ-lyon3.fr><br>>> Cc: "'koha'" <koha@lists.katipo.co.nz>, "'koha-devel'"<br>>> <koha-devel@lists.koha-community.org><br>>> Subject: Re: [Koha-devel] [Koha] OAI-PMH harvester<br>>> Message-ID: <07af01d8e8ca$dfbddef0$9f399cd0$@prosentient.com.au><br>>> Content-Type: text/plain; charset="utf-8"<br>>> <br>>> Hi Sonia,<br>>> <br>>> <br>>> <br>>> I’m excited to hear that KohaLA would like to finance an OAI-PMH <br>>> client in<br>>> Koha! This functionality is always brewing in the back of my mind, <br>>> since I<br>>> first raised 10662 back in 2013.<br>>> <br>>> <br>>> <br>>> As Tomas says, I think that the background jobs are a key component <br>>> for<br>>> processing incoming OAI-PMH records.<br>>> <br>>> <br>>> <br>>> However, the ***missing component right now is the scheduling of the<br>>> OAI-PMH harvesting tasks***, and I think this is where opinions get<br>>> divided. Below, I’ll provide some history and opinions on Koha <br>>> OAI-PMH.<br>>> <br>>> <br>>> <br>>> --<br>>> <br>>> <br>>> <br>>> With 10662, the sponsored goal was for Koha library staff to schedule<br>>> OAI-PMH harvests through the Web UI. However, Fridolin from BibLibre <br>>> raised<br>>> a point with me at Kohacon18 about how letting library staff control <br>>> the<br>>> timing of harvesting tasks could be a problem for support vendors. If <br>>> too<br>>> many libraries using the same public IP address tried to harvest from <br>>> the<br>>> same OAI-PMH repository, they could be rate limited or blocked. There <br>>> could<br>>> also be server load concerns. So there probably needs to be a balance<br>>> between user configuration and system configuration. If I recall <br>>> correctly,<br>>> this is how DSpace’s OAI-PMH harvester works. Users set up targets and <br>>> can<br>>> start/stop harvests, but things like frequency and concurrency are <br>>> handled<br>>> by the system configuration.<br>>> <br>>> <br>>> <br>>> Based on my experience working on OAI-PMH on and off for nearly 10 <br>>> years<br>>> and as a Koha support vendor, I think my preference would be for <br>>> sysadmins<br>>> to handle most of the OAI-PMH harvesting details.<br>>> <br>>> <br>>> <br>>> The sponsorship for 10662 had certain requirements that many other<br>>> libraries might not have, which is what made me think that it might be<br>>> better to have an external client that connects to Koha. I thought <br>>> maybe I<br>>> could get the ordinary requirements pushed into Koha, and then handle<br>>> extraordinary requirements externally. However, an external harvester <br>>> won’t<br>>> perform as fast as an internal harvester. (The compromise would be to <br>>> write<br>>> the harvester in such a way that people could provide different <br>>> OAI-PMH<br>>> harvester Perl modules that all stage records using the same core Koha<br>>> modules.)<br>>> <br>>> <br>>> <br>>> Even then… the scheduling would depend on a library’s needs. Back in <br>>> 2013,<br>>> I had a Koha OAI-PMH harvester which worked as a cronjob. It would run <br>>> each<br>>> night. However, some libraries want to run OAI-PMH harvests as <br>>> frequently<br>>> as every 3 seconds. A cronjob’s smallest frequency is 60 seconds, so <br>>> that<br>>> wouldn’t work for that requirement.<br>>> <br>>> <br>>> <br>>> If a cronjob isn’t suitable, then I think you’d need a daemon created <br>>> by a<br>>> new command like “koha-oai --start <instance_name>”. It could read a<br>>> configuration file and handle scheduling accordingly. With 10662, I <br>>> used<br>>> the POE module, because I knew it well and it has some timer tools for<br>>> scheduling tasks. If I were to work on it again, I’d probably use<br>>> Mojo::IOLoop instead these days, since Mojolicious is already part of <br>>> Koha<br>>> while POE is not. (That said, using modules like Mojo and POE are<br>>> difficult, because they’re difficult to test using automation. That <br>>> was one<br>>> of the stumbling blocks with 10662. While the 10662 harvester worked <br>>> very<br>>> well, it was difficult to unit test. In hindsight, I should’ve written <br>>> it<br>>> in a way that was easier to unit test, but it had a lot of <br>>> event-driven<br>>> code which made things more difficult.)<br>>> <br>>> <br>>> <br>>> Another option would be to create a generic daemon for task scheduling <br>>> in<br>>> general (e.g. “koha-schedule”). Koha could use this for many things, <br>>> but<br>>> it’s a project in itself.<br>>> <br>>> <br>>> <br>>> --<br>>> <br>>> <br>>> <br>>> The process of downloading OAI-PMH records and importing MARCXML into <br>>> Koha<br>>> is actually a fairly straightforward process. The difficulty is the <br>>> task<br>>> scheduling and management of tasks (and unit testing).<br>>> <br>>> <br>>> <br>>> I don’t know the answer that will make everyone happy. There’s lots of<br>>> different ways of managing and scheduling the tasks. Based on my<br>>> experience, I’d suggest targeting the simplest approach first, because<br>>> complexity will make it less likely for the project to succeed.<br>>> <br>>> <br>>> <br>>> On that note, I’d be happy to test/QA any OAI-PMH harvester put <br>>> forward.<br>>> When I was writing OAI-PMH harvester patches, I found it really hard <br>>> to get<br>>> QA, so I’m happy to be that resource for someone else. I’ve spent a <br>>> lot of<br>>> time thinking about this topic, so happy to provide advice, warnings,<br>>> emotional support 😉.<br>>> <br>>> <br>>> <br>>> David Cook<br>>> <br>>> Senior Software Engineer<br>>> <br>>> Prosentient Systems<br>>> <br>>> Suite 7.03<br>>> <br>>> 6a Glen St<br>>> <br>>> Milsons Point NSW 2061<br>>> <br>>> Australia<br>>> <br>>> <br>>> <br>>> Office: 02 9212 0899<br>>> <br>>> Online: 02 8005 0595<br>>> <br>>> <br>>> <br>>> From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On <br>>> Behalf<br>>> Of Tomas Cohen Arazi<br>>> Sent: Wednesday, 26 October 2022 3:46 AM<br>>> To: BOUIS Sonia <sonia.bouis@univ-lyon3.fr><br>>> Cc: koha <koha@lists.katipo.co.nz>; koha-devel <<br>>> koha-devel@lists.koha-community.org><br>>> Subject: Re: [Koha-devel] [Koha] OAI-PMH harvester<br>>> <br>>> <br>>> <br>>> I think with background jobs we have most of the framework that is <br>>> needed<br>>> to deal with this within Koha.<br>>> <br>>> <br>>> <br>>> Best regards<br>>> <br>>> <br>>> <br>>> El mar, 25 oct 2022 7:08, BOUIS Sonia <sonia.bouis@univ-lyon3.fr <br>>> <mailto:<br>>> sonia.bouis@univ-lyon3.fr> > escribió:<br>>> <br>>> Hi,<br>>> KohaLA would like to finance an OAI-PMH client in Koha but, we have<br>>> questions that we want to raise to the community.<br>>> There was already tries to propose an OAI-PMH client :<br>>> - https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 : <br>>> it's<br>>> an old project that doesnt seem compatible with the current version of <br>>> Koha<br>>> - https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25905 : <br>>> the<br>>> scope is more to use an external OAI-PMH client and to connect it to <br>>> Koha<br>>> <br>>> Our main question is about the way to handle this. Do you think that <br>>> it's<br>>> a better idea to use an external software or PERL routine and to find <br>>> a way<br>>> to connect it to Koha. Or would it be better to a new module in Koha <br>>> from<br>>> scratch and that Koha have his own OAI-PMH client.<br>>> <br>>> Please, let us hear your toughts about this projet.<br>>> <br>>> Kind regards<br>>> <br>>> Sonia<br>>> <br>>> Sonia BOUIS<br>>> ------------------------------------------------------<br>>> Responsable du Service informatique documentaire Département d'Appui à <br>>> la<br>>> Recherche et aux Projets (DARP) Bibliothèques universitaires <br>>> Université<br>>> Jean Moulin Lyon 3 ADRESSE GÉOGRAPHIQUE > Manufacture des Tabacs | 6 <br>>> cours<br>>> Albert Thomas | LYON 8e ADRESSE POSTALE > Bibliothèque de la <br>>> Manufacture |<br>>> 1C avenue des Frères Lumière | CS 78242 - 69372 LYON CEDEX 08<br>>> <br>>> Ligne directe : 33 (0)4 78 78 79 03<br>>> <br>>> http://bu.univ-lyon3.fr<http://bu.univ-lyon3.fr/>| Suivez-nous > <br>>> Facebook<<br>>> https://www.facebook.com/bulyon3/> | <br>>> Twitter<https://twitter.com/bulyon3>|<br>>> Instagram<https://www.instagram.com/bu.lyon3/?hl=fr><br>>> <br>>> _______________________________________________<br>>> <br>>> Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz<br>>> <mailto:Koha@lists.katipo.co.nz><br>>> Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha<br>>> <br>>> -------------- next part --------------<br>>> An HTML attachment was scrubbed...<br>>> URL: <<br>>> http://lists.koha-community.org/pipermail/koha-devel/attachments/20221026/d7712779/attachment-0001.htm<br>>> ><br>>> <br>>> ------------------------------<br>>> <br>>> Subject: Digest Footer<br>>> <br>>> _______________________________________________<br>>> Koha-devel mailing list<br>>> Koha-devel@lists.koha-community.org<br>>> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel<br>>> website : https://www.koha-community.org/ git :<br>>> https://git.koha-community.org/ bugs : <br>>> https://bugs.koha-community.org/<br>>> <br>>> <br>>> ------------------------------<br>>> <br>>> End of Koha-devel Digest, Vol 203, Issue 15<br>>> *******************************************<br>>> _______________________________________________<br>>> <br>>> Koha mailing list http://koha-community.org<br>>> Koha@lists.katipo.co.nz<br>>> Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha<br>>> <br>> _______________________________________________<br>> <br>> Koha mailing list http://koha-community.org<br>> Koha@lists.katipo.co.nz<br>> Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha<br><br>-- <br>Arthur Suzuki, 🌈🏔️<br>Développeur @BibLibre<br><br><br>------------------------------<br><br>Subject: Digest Footer<br><br>_______________________________________________<br>Koha-devel mailing list<br>Koha-devel@lists.koha-community.org<br>https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel<br>website : https://www.koha-community.org/<br>git : https://git.koha-community.org/<br>bugs : https://bugs.koha-community.org/<br><br><br>------------------------------<br><br>End of Koha-devel Digest, Vol 204, Issue 17<br>*******************************************<br></div><br><br>NOTE: You are receiving this mail because, the Requester/Technician wanted you to get notified on this request creation.<br></body></html>