December 2022 - Koha-devel - lists.koha-community.org

How to make self-referential URLs in Koha
by dcook＠prosentient.com.au 30 Jan '23

30 Jan '23

Hi all, With some recent code, I've been using creating self-referential URL in Koha using CGI->url(), and passing it as a value. With CGI, the URL is correct (ie starting with /cgi-bin/koha). With Plack, it's not, because the OPAC is mounted as "/opac". I worked around this by rewriting the self-referential URL in my code, but I was wondering if anyone else had any thoughts on this. I've noticed there are a few places in Koha where we use CGI->url() and I'm guessing we may have unexpected bugs in Plack-enabled Koha as a result. Of course, one workaround would be to have /opac and /intranet aliases or URL rewriting in Apache, so that the self-referential URLs created by Plack still end up resolving. I wonder if that's the easiest path forward. In the past, I have wondered if we could get rid of "/cgi-bin/koha" as a path prefix all together (especially as cgi-bin scares security tools), but it's so deeply embedded in Koha at this point that it seems a huge task to replace it. Anyway, just throwing that out there. David Cook Systems Librarian Prosentient Systems 72/330 Wattle St Ultimo, NSW 2007 Australia Office: 02 9212 0899 Online: 02 8005 0595

6 8

Re: [Koha-devel] [Koha] OAI-PMH harvester
by Arthur Suzuki 23 Jan '23

23 Jan '23

Hello there, If I may suggest a good harvester library, Catmandu may do the job pretty well. I've not used the OAI module but used it to harvest from a JSON source and transform to an UNIMARC file with pretty good success so far. It can export seamlessly to iso2709 or marcxml. https://metacpan.org/dist/Catmandu-OAI Best, Arthur On 2022-11-22 15:57, Mike D. wrote: > Hey. Hey, > I'm really glad to see the OAI-PMH harvester debate going on for Koha. > I > think if we choose a good external harvester with support, we can save > a > lot of energy and resources to implement related activities in the > system. > Shoveling the logs is only part of the story. The easy part. Since the > result of shoveling is a lot of records, most of the time we can't > avoid > post-processing, merging with the records in the local database. For > example, if you need to update records from a source where there are > millions of records, but there are hundreds of thousands in the local > database. Only a slice of that huge amount is relevant. If we design > the > processing workflow wrong, it will take unnecessarily long and burn > valuable resources. > I would hereby like to invite us to be in touch, to debate and share > our > experiences. Let's get this area moving towards a successful finish. > > Take care. > > Michal > > út 22. 11. 2022 v 15:13 odesílatel BOUIS Sonia > <sonia.bouis(a)univ-lyon3.fr> > napsal: > >> Hi, >> Thanks to David, Tomas, Michal and Michael for your replies. >> >> So we have decided to evaluate several external OAI-PMH client that >> could >> be used by Koha and to choose one in the end of January >> There a lot to do after that and we discussed about the background >> jobs >> and cronjobs seems to be appropriate. We thought that the settings in >> the >> koha intranet should be only to define URLs, SETs, or XSLT sheets (for >> example, to transform DC XML in MARCXML). >> >> We are only at the begining of the process 😊 >> >> Kind regards, >> Sonia >> >> ------------------------------ >> >> Message: 2 >> Date: Wed, 26 Oct 2022 10:37:49 +1100 >> From: "David Cook" <dcook(a)prosentient.com.au> >> To: "'Tomas Cohen Arazi'" <tomascohen(a)gmail.com>, "'BOUIS Sonia'" >> <sonia.bouis(a)univ-lyon3.fr> >> Cc: "'koha'" <koha(a)lists.katipo.co.nz>, "'koha-devel'" >> <koha-devel(a)lists.koha-community.org> >> Subject: Re: [Koha-devel] [Koha] OAI-PMH harvester >> Message-ID: <07af01d8e8ca$dfbddef0$9f399cd0$(a)prosentient.com.au> >> Content-Type: text/plain; charset="utf-8" >> >> Hi Sonia, >> >> >> >> I’m excited to hear that KohaLA would like to finance an OAI-PMH >> client in >> Koha! This functionality is always brewing in the back of my mind, >> since I >> first raised 10662 back in 2013. >> >> >> >> As Tomas says, I think that the background jobs are a key component >> for >> processing incoming OAI-PMH records. >> >> >> >> However, the ***missing component right now is the scheduling of the >> OAI-PMH harvesting tasks***, and I think this is where opinions get >> divided. Below, I’ll provide some history and opinions on Koha >> OAI-PMH. >> >> >> >> -- >> >> >> >> With 10662, the sponsored goal was for Koha library staff to schedule >> OAI-PMH harvests through the Web UI. However, Fridolin from BibLibre >> raised >> a point with me at Kohacon18 about how letting library staff control >> the >> timing of harvesting tasks could be a problem for support vendors. If >> too >> many libraries using the same public IP address tried to harvest from >> the >> same OAI-PMH repository, they could be rate limited or blocked. There >> could >> also be server load concerns. So there probably needs to be a balance >> between user configuration and system configuration. If I recall >> correctly, >> this is how DSpace’s OAI-PMH harvester works. Users set up targets and >> can >> start/stop harvests, but things like frequency and concurrency are >> handled >> by the system configuration. >> >> >> >> Based on my experience working on OAI-PMH on and off for nearly 10 >> years >> and as a Koha support vendor, I think my preference would be for >> sysadmins >> to handle most of the OAI-PMH harvesting details. >> >> >> >> The sponsorship for 10662 had certain requirements that many other >> libraries might not have, which is what made me think that it might be >> better to have an external client that connects to Koha. I thought >> maybe I >> could get the ordinary requirements pushed into Koha, and then handle >> extraordinary requirements externally. However, an external harvester >> won’t >> perform as fast as an internal harvester. (The compromise would be to >> write >> the harvester in such a way that people could provide different >> OAI-PMH >> harvester Perl modules that all stage records using the same core Koha >> modules.) >> >> >> >> Even then… the scheduling would depend on a library’s needs. Back in >> 2013, >> I had a Koha OAI-PMH harvester which worked as a cronjob. It would run >> each >> night. However, some libraries want to run OAI-PMH harvests as >> frequently >> as every 3 seconds. A cronjob’s smallest frequency is 60 seconds, so >> that >> wouldn’t work for that requirement. >> >> >> >> If a cronjob isn’t suitable, then I think you’d need a daemon created >> by a >> new command like “koha-oai --start <instance_name>”. It could read a >> configuration file and handle scheduling accordingly. With 10662, I >> used >> the POE module, because I knew it well and it has some timer tools for >> scheduling tasks. If I were to work on it again, I’d probably use >> Mojo::IOLoop instead these days, since Mojolicious is already part of >> Koha >> while POE is not. (That said, using modules like Mojo and POE are >> difficult, because they’re difficult to test using automation. That >> was one >> of the stumbling blocks with 10662. While the 10662 harvester worked >> very >> well, it was difficult to unit test. In hindsight, I should’ve written >> it >> in a way that was easier to unit test, but it had a lot of >> event-driven >> code which made things more difficult.) >> >> >> >> Another option would be to create a generic daemon for task scheduling >> in >> general (e.g. “koha-schedule”). Koha could use this for many things, >> but >> it’s a project in itself. >> >> >> >> -- >> >> >> >> The process of downloading OAI-PMH records and importing MARCXML into >> Koha >> is actually a fairly straightforward process. The difficulty is the >> task >> scheduling and management of tasks (and unit testing). >> >> >> >> I don’t know the answer that will make everyone happy. There’s lots of >> different ways of managing and scheduling the tasks. Based on my >> experience, I’d suggest targeting the simplest approach first, because >> complexity will make it less likely for the project to succeed. >> >> >> >> On that note, I’d be happy to test/QA any OAI-PMH harvester put >> forward. >> When I was writing OAI-PMH harvester patches, I found it really hard >> to get >> QA, so I’m happy to be that resource for someone else. I’ve spent a >> lot of >> time thinking about this topic, so happy to provide advice, warnings, >> emotional support 😉. >> >> >> >> David Cook >> >> Senior Software Engineer >> >> Prosentient Systems >> >> Suite 7.03 >> >> 6a Glen St >> >> Milsons Point NSW 2061 >> >> Australia >> >> >> >> Office: 02 9212 0899 >> >> Online: 02 8005 0595 >> >> >> >> From: Koha-devel <koha-devel-bounces(a)lists.koha-community.org> On >> Behalf >> Of Tomas Cohen Arazi >> Sent: Wednesday, 26 October 2022 3:46 AM >> To: BOUIS Sonia <sonia.bouis(a)univ-lyon3.fr> >> Cc: koha <koha(a)lists.katipo.co.nz>; koha-devel < >> koha-devel(a)lists.koha-community.org> >> Subject: Re: [Koha-devel] [Koha] OAI-PMH harvester >> >> >> >> I think with background jobs we have most of the framework that is >> needed >> to deal with this within Koha. >> >> >> >> Best regards >> >> >> >> El mar, 25 oct 2022 7:08, BOUIS Sonia <sonia.bouis(a)univ-lyon3.fr >> <mailto: >> sonia.bouis(a)univ-lyon3.fr> > escribió: >> >> Hi, >> KohaLA would like to finance an OAI-PMH client in Koha but, we have >> questions that we want to raise to the community. >> There was already tries to propose an OAI-PMH client : >> - https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 : >> it's >> an old project that doesnt seem compatible with the current version of >> Koha >> - https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25905 : >> the >> scope is more to use an external OAI-PMH client and to connect it to >> Koha >> >> Our main question is about the way to handle this. Do you think that >> it's >> a better idea to use an external software or PERL routine and to find >> a way >> to connect it to Koha. Or would it be better to a new module in Koha >> from >> scratch and that Koha have his own OAI-PMH client. >> >> Please, let us hear your toughts about this projet. >> >> Kind regards >> >> Sonia >> >> Sonia BOUIS >> ------------------------------------------------------ >> Responsable du Service informatique documentaire Département d'Appui à >> la >> Recherche et aux Projets (DARP) Bibliothèques universitaires >> Université >> Jean Moulin Lyon 3 ADRESSE GÉOGRAPHIQUE > Manufacture des Tabacs | 6 >> cours >> Albert Thomas | LYON 8e ADRESSE POSTALE > Bibliothèque de la >> Manufacture | >> 1C avenue des Frères Lumière | CS 78242 - 69372 LYON CEDEX 08 >> >> Ligne directe : 33 (0)4 78 78 79 03 >> >> http://bu.univ-lyon3.fr<http://bu.univ-lyon3.fr/>| Suivez-nous > >> Facebook< >> https://www.facebook.com/bulyon3/> | >> Twitter<https://twitter.com/bulyon3>| >> Instagram<https://www.instagram.com/bu.lyon3/?hl=fr> >> >> _______________________________________________ >> >> Koha mailing list http://koha-community.org Koha(a)lists.katipo.co.nz >> <mailto:Koha@lists.katipo.co.nz> >> Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha >> >>

2 1

Koha releases are available 🎁
by Mason James 16 Jan '23

16 Jan '23

kia ora community, the latest Koha packages and tarballs are ready release notes are here... https://koha-community.org/koha-21-11-03-released https://koha-community.org/koha-21-05-11-released https://koha-community.org/koha-20-11-15-released https://koha-community.org/koha-19-11-26-released cheers, Mason

2 2

How best to support MARC21 880?
by David Cook 01 Jan '23

01 Jan '23

Hi all, The MARC21 880 field allows you to provide alternative graphical representations for pretty much any other MARC field. Typically, in Koha (and perhaps other library systems), we display these alternative graphical representations on the "detail page". You can see the roman script, the Arabic script, the Chinese script, etc. In Koha, we also show some data in the search results, and that's more problematic. because the search results should take up a limited amount of space on the screen, but hypothetically you can include many different scripts via the 880. I used to work in a government library in Canada where many of our bibliographic records contained quite a few different 880s. So it does happen. I have a library that has catalogued records using the roman alphabet (ie using Pinyin) and provided Chinese versions via the 880. They'd like to show both the romanized and Chinese versions throughout Koha (e.g. in the cart, checkout screens, course reserves, etc.) I've coded some of those customizations, since they only have 880s in Chinese, but I'm increasingly reluctant to make these changes. I don't want to carry these customizations locally, but I don't know that the Koha community would accept these changes. What do you all think about this? Should we include every 880 title and author in every screen? Should we maybe include them but hide them by default using CSS and then allow certain libraries to just show particular languages used by their patrons or by checking the current language? I know I can go faster on my own, but I rather go farther with all of you, so please let me know your thoughts : ) David Cook Senior Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595

2 1

Koha Community Newsletter: December 2022
by Koha Newsletter 29 Dec '22

29 Dec '22

The Koha Community Newsletter for December 2022 is here: https://koha-community.org/koha-community-newsletter-december-2022/ Many thanks to everyone who submitted articles and news to this newsletter! Please feel free to email me with any corrections or suggestions. -- Michael Kuhn Editor, Koha Community Newsletter

1 0

KTD - stable branches
by Tomas Cohen Arazi 23 Dec '22

23 Dec '22

Hi all, I made some pushes to the stable branches (22.11, 22.05 and 21.11) so the images are (internally) aligned with (master's) `ktd` command. Please let me know if you encounter any issues. Merry xmas everyone! -- Tomás Cohen Arazi Theke Solutions (http://theke.io) ✆ +54 9351 3513384 GPG: B2F3C15F

1 0

The many failings of background_jobs_worker.pl
by Philippe Blouin 22 Dec '22

22 Dec '22

Howdy! Since moving a lot of our users to 22.05.06, we've installed the worker everywhere. But the number of issues encountered is staggering. The first one was Can't call method "process" on an undefined value where the id received from MQ was not found in the DB, and the process is going straight to process_job and failing. Absolutely no idea how that occurs, seems completely counterintuitive (the ID comes from the DB after all), but here it is. Hacked the code to add a "sleep 1" to fix most of that one. Then came the fact that stored events were not checked if the connection to MQ was successful at startup. Bug 30654 refers it. Hacked a little "$init" in there to clear that up at startup. Then came the malformed UTF-8 character in JSON string, at character offset 296 (before "\x{e9}serv\x{e9} au ...") at decode_json that crashes the whole process. And for some reason, it never gets over it, gets the same problem at every restart, like the event is never "eaten" from the queue. Hacked an eval then a try-catch over it... After coding a monitor to alert when a background_jobs has been "new" over 5 minutes in the DB, I was inundated by messages. There's alway one elasticsearch_update that escapes among the flurry, and they slowly add up. At this point, the only viable solution is to run the workers but disable RabbitMQ everywhere. Are we really the only ones experiencing that? Regards, PS Our servers are well-above-average Debian 11 machines with lot of firepower (ram, cpu, i/o...). -- Philippe Blouin, Directeur de la technologie Tél. : (833) 465-4276, poste 230 philippe.blouin(a)inLibro.com inLibro | pour esprit libre | www.inLibro.com <http://www.inLibro.com>

6 12

Added Testing SSO guide to wiki
by David Cook 20 Dec '22

20 Dec '22

Hi all, At Martin's request at https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31651#c3 I've put together a wiki page for testing single sign on (SSO) using Keycloak in koha-testing-docker: https://wiki.koha-community.org/wiki/Testing_SSO#SAML It's a first draft, so it might not be perfect, but I was able to use Keycloak to log into Koha using OpenID Connect and SAML while writing the guides, so it should work. Note that the instructions are just for testing. They're not production instructions. The intention is just to provide this for Koha developers, so that everyone can easily work on Koha SSO patches without being a SSO expert or without having commercial 3rd party SSO services available. I hope people find it useful! Let me know if you have any feedback on how it can improved. (Note: I already know the formatting is awful, but I was in a hurry and I'm not familiar enough with Mediawiki to format it better. Maybe someone can give a hand there?) David Cook Senior Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595

1 0

HOLDPLACED should be emailed to item branch rather than patron branch
by David Cook 19 Dec '22

19 Dec '22

Hi all, I had some librarians raise to me that the HOLDPLACED email goes to the patron branch rather than the item branch. I've raised https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32499 to address this, but can anyone think of a reason why it should stay the patron branch and not the item branch? David Cook Senior Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595

1 0

ktd - enhancements
by Tomas Cohen Arazi 19 Dec '22

19 Dec '22

Hi all, just wanted to let you know that we solved a long standing issue in KTD [1] that made it awkward to switch from using ES or not with KTD (required tweaking the .env file and would break if things didn't match). The `ktd` command now takes advantage of this enhancement so, as a reminder, if you want to run KTD with ES 7 [2] you just need to run: $ ktd --es7 up Any questions about it, just ask on IRC or on this thread. Best regards [1] https://gitlab.com/koha-community/koha-testing-docker/-/issues/306 [2] ktd --help will show you all the available options -- Tomás Cohen Arazi Theke Solutions (http://theke.io) ✆ +54 9351 3513384 GPG: B2F3C15F

1 0