<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"><html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body>A new request with request id 18741 has been created by koha-devel-request@lists.koha-community.org. Short info on the request is : <br><br>Title : Koha-devel Digest, Vol 185, Issue 21<br>Category : <br>Description : <div>Send Koha-devel mailing list submissions to<br> koha-devel@lists.koha-community.org<br><br>To subscribe or unsubscribe via the World Wide Web, visit<br> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel<br>or, via email, send a message with subject or body 'help' to<br> koha-devel-request@lists.koha-community.org<br><br>You can reach the person managing the list at<br> koha-devel-owner@lists.koha-community.org<br><br>When replying, please edit your Subject line so it is more specific<br>than "Re: Contents of Koha-devel digest..."<br><br><br>Today's Topics:<br><br> 1. Re: Optimizing Starman startup (dcook@prosentient.com.au)<br> 2. Re: Optimizing Starman startup (dcook@prosentient.com.au)<br> 3. bulkmarcimport.pl ... re-import/update existing records<br> (David Schmidt)<br><br><br>----------------------------------------------------------------------<br><br>Message: 1<br>Date: Thu, 22 Apr 2021 09:14:31 +1000<br>From: <dcook@prosentient.com.au><br>To: "'Julian Maurice'" <julian.maurice@biblibre.com>,<br> <koha-devel@lists.koha-community.org><br>Subject: Re: [Koha-devel] Optimizing Starman startup<br>Message-ID: <030f01d73704$15f8e930$41eabb90$@prosentient.com.au><br>Content-Type: text/plain; charset="UTF-8"<br><br>It's too bad that we're not using Unix sockets out of the box (like we do with Starman) as that would make Bug 26625 less pressing but maybe I'll look more at that...<br><br>The only downside with starting Koha as a Mojolicious application at the moment seems to be the experimentalness of it. I'm wondering how we can start a paradigm shift towards using it for production for everyone globally...<br><br>David Cook<br>Software Engineer<br>Prosentient Systems<br>Suite 7.03<br>6a Glen St<br>Milsons Point NSW 2061<br>Australia<br><br>Office: 02 9212 0899<br>Online: 02 8005 0595<br><br>-----Original Message-----<br>From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of Julian Maurice<br>Sent: Wednesday, 21 April 2021 7:00 PM<br>To: koha-devel@lists.koha-community.org<br>Subject: Re: [Koha-devel] Optimizing Starman startup<br><br>Le 21/04/2021 à 09:26, dcook@prosentient.com.au a écrit :<br>> I recall someone (Julian or Kyle?) did some work to make Koha into a <br>> Mojolicious application<br><br>Yes, since 20.11 Koha can be started as a Mojolicious application.<br><br>`hypnotoad bin/intranet`<br><br>But you'll probably need<br>https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26625 in order to configure the listen port/address.<br><br>--<br>Julian Maurice<br>BibLibre<br>_______________________________________________<br>Koha-devel mailing list<br>Koha-devel@lists.koha-community.org<br>https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel<br>website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/<br><br><br><br><br>------------------------------<br><br>Message: 2<br>Date: Thu, 22 Apr 2021 09:23:59 +1000<br>From: <dcook@prosentient.com.au><br>To: "'Ere Maijala'" <ere.maijala@helsinki.fi>,<br> <koha-devel@lists.koha-community.org><br>Subject: Re: [Koha-devel] Optimizing Starman startup<br>Message-ID: <031001d73705$6854efc0$38fecf40$@prosentient.com.au><br>Content-Type: text/plain; charset="UTF-8"<br><br>Hi Ere,<br><br>I think you're right about the refs. While they get resolved by the OpenAPI plugin, you probably have to resolve them before trying to dynamically inject the routes from plugins. <br><br>Jan Thorsen (the author of Mojolicious::Plugin::OpenAPI and JSON::Validator) thinks that the ref resolution is actually what's taking so long. I looked it up and I think we have over 400 different references in the main OpenAPI spec alone. I haven't profiled it but something to think about.<br><br>At some point, I'm going to have a play with newer versions of the modules. I'm gong to look at Ubuntu 20.04 and newer Debian versions to see what I can get away with in terms of newness. Needs more investigation, but I am really hoping that this is an issue that can be solved by just upgrading the OS. <br><br>I find Jan's code to be unnecessarily opaque (could use more descriptive comments and function naming) but... I'll investigate. Probably not right away as I have a bunch of other priorities that I have to address but... this is on my mind.<br><br>Starman startup time is probably the thing about Koha annoying me the most right now and probably the most practical thing I can improve at the moment...<br><br>David Cook<br>Software Engineer<br>Prosentient Systems<br>Suite 7.03<br>6a Glen St<br>Milsons Point NSW 2061<br>Australia<br><br>Office: 02 9212 0899<br>Online: 02 8005 0595<br><br>-----Original Message-----<br>From: Ere Maijala <ere.maijala@helsinki.fi> <br>Sent: Wednesday, 21 April 2021 6:31 PM<br>To: dcook@prosentient.com.au; koha-devel@lists.koha-community.org<br>Subject: Re: [Koha-devel] Optimizing Starman startup<br><br>Hi David,<br><br>I wish I'd remember all the details, but my memory fails me. I think not using JSON had something to do with how the refs are resolved. That may or may not have been the reason, but if everything works with JSON module, I can't think of a reason not to use it.<br><br>Thanks for taking a look!<br><br>--Ere<br><br>dcook@prosentient.com.au kirjoitti 21.4.2021 klo 3.28:<br>> Hi Ere,<br>> <br>> Thanks for your reply. 24700 looks much better. I'll look at backporting it locally.<br>> <br>> Although I'm looking at JSON::Validator::OpenAPI::Mojolicious at https://metacpan.org/pod/release/JHTHORSEN/Mojolicious-Plugin-OpenAPI-2.19/lib/JSON/Validator/OpenAPI/Mojolicious.pm and it says "Do not use this module directly. Use Mojolicious::Plugin::OpenAPI instead." I notice that you're using the "bundle" method. Do we really need that there? Why don't we just load the JSON using the JSON module, merge with the plugin spec files, and then pass it to the OpenAPI plugin? Shouldn't the plugin take care of the $ref replacement?<br>> <br>> Hmm... I didn't realize until now that the OpenAPI plugin was doing a validate behind the scenes. That's tricky.<br>> <br>> At a glance, we might be able to pre-load the app into the Starman <br>> master process pre-fork. There are warnings about doing that with open <br>> database connections, so we'd need to review plack.psgi, but a quick <br>> glance suggests it might be OK. (Alternatively, I have wondered about <br>> running the REST API as a separate process apart from Starman using <br>> hypnotoad. According to <br>> https://docs.mojolicious.org/Mojolicious/Guides/Cookbook, <br>> Mojo::Server::Prefork preloads the application in the manager/master <br>> process, and Hypnotoad is based off that, so that would help.)<br>> <br>> It does seem like changes to the OpenAPI plugin would be needed for caching.<br>> <br>> I'm going to try backporting your change and try pre-loading and see how far that gets me.<br>> <br>> David Cook<br>> Software Engineer<br>> Prosentient Systems<br>> Suite 7.03<br>> 6a Glen St<br>> Milsons Point NSW 2061<br>> Australia<br>> <br>> Office: 02 9212 0899<br>> Online: 02 8005 0595<br>> <br>> -----Original Message-----<br>> From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On <br>> Behalf Of Ere Maijala<br>> Sent: Tuesday, 20 April 2021 4:48 PM<br>> To: koha-devel@lists.koha-community.org<br>> Subject: Re: [Koha-devel] Optimizing Starman startup<br>> <br>> Hi,<br>> <br>> I did some work on improving it here:<br>> <br>> https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24700<br>> <br>> That shaved a good bit of time from it, but it's still a heavy <br>> operation, and it would make sense to<br>> <br>> 1.) avoid doing it too often<br>> <br>> 2.) cache the results and avoid doing it if results are cached<br>> <br>> If you could address the first one, that'd go a long way. I'm afraid the second one would require changes to the OpenAPI plugin to support caching.<br>> <br>> --Ere<br>> <br>> dcook@prosentient.com.au kirjoitti 20.4.2021 klo 6.15:<br>>> Hi all,<br>>><br>>> Do you despair when you see the following periodically in “top” when <br>>> a starman worker is recreated ?<br>>><br>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+<br>>> COMMAND<br>>><br>>> 9529 my-koha 20 0 460108 197212 17172 R 100.0 0.4 0:03.41<br>>> /usr/share/koha/api/v1/app.pl<br>>><br>>> Or the following in top when you install koha-common package or <br>>> restart the koha-common service?<br>>><br>>> 11101 1-koha 20 0 447232 193320 16076 R 10.6 0.4 0:09.09<br>>> /usr/share/koha/api/v1/app.pl<br>>><br>>> 11168 1-koha 20 0 447240 193264 16056 R 10.6 0.4 0:08.72<br>>> /usr/share/koha/api/v1/app.pl<br>>><br>>> 11306 2-koha 20 0 447220 193148 16000 R 10.6 0.4 0:08.07<br>>> /usr/share/koha/api/v1/app.pl<br>>><br>>> 11543 2-koha 20 0 447232 193036 15828 R 10.6 0.4 0:07.07<br>>> /usr/share/koha/api/v1/app.pl<br>>><br>>> 11784 3-koha 20 0 441536 189664 16172 R 10.6 0.4 0:06.04<br>>> /usr/share/koha/api/v1/app.pl<br>>><br>>> 11830 3-koha 20 0 439548 187212 15748 R 10.6 0.4 0:05.82<br>>> /usr/share/koha/api/v1/app.pl<br>>><br>>> 11831 4-koha 20 0 438620 186344 15748 R 10.6 0.4 0:05.81<br>>> /usr/share/koha/api/v1/app.pl<br>>><br>>> 11853 4-koha 20 0 437680 185672 16000 R 10.6 0.4 0:05.79<br>>> /usr/share/koha/api/v1/app.pl<br>>><br>>> Well, I still have a lot of investigation left to do, but I notice 1 <br>>> place that a lot of time taken is here (per worker):<br>>><br>>> my $validator = JSON::Validator::OpenAPI::Mojolicious->new;<br>>><br>>> $validator->load_and_validate_schema(<br>>><br>>> $self->home->rel_file("api/v1/swagger/swagger.json"),<br>>><br>>> {<br>>><br>>> allow_invalid_ref => 1,<br>>><br>>> }<br>>><br>>> );<br>>><br>>> push @{$self->routes->namespaces}, 'Koha::Plugin';<br>>><br>>> my $spec = $validator->schema->data;<br>>><br>>> $self->plugin(<br>>><br>>> 'Koha::REST::Plugin::PluginRoutes' => {<br>>><br>>> spec => $spec,<br>>><br>>> validator => $validator<br>>><br>>> }<br>>><br>>> );<br>>><br>>> $self->plugin(<br>>><br>>> OpenAPI => {<br>>><br>>> spec => $spec,<br>>><br>>> route =><br>>> $self->routes->under('/api/v1')->to('Auth#under'),<br>>><br>>> allow_invalid_ref =><br>>><br>>> 1, # required by our spec because $ref directly<br>>> under<br>>><br>>> # Paths-, Parameters-, Definitions- & <br>>> Info-object<br>>><br>>> # is not allowed by the OpenAPI specification.<br>>><br>>> }<br>>><br>>> );<br>>><br>>> Anyone have ideas for improving this? Do we have to validate the <br>>> schema every time? Can we move the schema validation into a different <br>>> module and preload it into Starman using the -M flag so that it’s <br>>> done<br>>> 1 time per Starman master instance rather than 1 time per Starman worker instance?<br>>><br>>> I find “/usr/share/koha/api/v1/app.pl” to be the bane of deployments, <br>>> as it puts a massive load on a server, when you have multiple Koha <br>>> instances on the server.<br>>><br>>> David Cook<br>>><br>>> Software Engineer<br>>><br>>> Prosentient Systems<br>>><br>>> Suite 7.03<br>>><br>>> 6a Glen St<br>>><br>>> Milsons Point NSW 2061<br>>><br>>> Australia<br>>><br>>> Office: 02 9212 0899<br>>><br>>> Online: 02 8005 0595<br>>><br>>><br>>> _______________________________________________<br>>> Koha-devel mailing list<br>>> Koha-devel@lists.koha-community.org<br>>> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel<br>>> website : https://www.koha-community.org/ git :<br>>> https://git.koha-community.org/ bugs :<br>>> https://bugs.koha-community.org/<br>>><br>> <br>> --<br>> Ere Maijala<br>> Kansalliskirjasto / The National Library of Finland <br>> _______________________________________________<br>> Koha-devel mailing list<br>> Koha-devel@lists.koha-community.org<br>> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel<br>> website : https://www.koha-community.org/ git : <br>> https://git.koha-community.org/ bugs : <br>> https://bugs.koha-community.org/<br>> <br>> <br><br>--<br>Ere Maijala<br>Kansalliskirjasto / The National Library of Finland<br><br><br><br><br>------------------------------<br><br>Message: 3<br>Date: Thu, 22 Apr 2021 11:02:29 +0200<br>From: "David Schmidt" <mail@davidschmidt.at><br>To: Koha-devel@lists.koha-community.org<br>Subject: [Koha-devel] bulkmarcimport.pl ... re-import/update existing<br> records<br>Message-ID: <5ccd0889-5a24-41a1-9aca-b639e4330f97@www.fastmail.com><br>Content-Type: text/plain; charset="utf-8"<br><br>Hello koha folks,<br><br>###<br>I hope this is the correct list for this kind of email<br>###<br><br><br>We are currently exporting about 500k recods from a dabis system into a new koha installation.<br><br>the export file (MARCXML) goes through some transformations using catmandu.<br><br>I imported the first 25k records using this command:<br><br>```<br>bulkmarcimport.pl -M MARCXML -file tmp/marc_001.xml<br>```<br><br>after inspecting the records some of the catmandu fixes had to be updated and now im trying to update the existing koha records with the new marc_002.xml file.<br><br>```<br>bulkmarcimport.pl -M MARCXML -update -isbn -file tmp/marc_002.xml<br>```<br><br>this gives me alot of these errors:<br>```<br>WARNING: Updating record failed at /usr/share/koha/bin/migration_tools/bulkmarcimport.pl line 470, <GEN24> line 190.<br>..Use of uninitialized value $originalid in concatenation (.) or string at /usr/share/koha/bin/migration_tools/bulkmarcimport.pl line 470, <GEN24> line 194.<br>WARNING: Updating record failed at /usr/share/koha/bin/migration_tools/bulkmarcimport.pl line 470, <GEN24> line 194.<br>```<br><br>I also tried using -match but I havent figured out how the parameters for -match are supposed to look like<br>```<br>bulkmarcimport.pl -m MARCXML -update -match="001" -file tmp/marc_002.xml<br>```<br><br>```<br>.Use of uninitialized value $recorddata in pattern match (m//) at /usr/share/koha/bin/migration_tools/bulkmarcimport.pl line 603, <GEN24> line 2.<br>unable to search the database for duplicates : No query entered at /usr/share/koha/bin/migration_tools/bulkmarcimport.pl line 323, <GEN24> line 2.<br>```<br><br>help or pointers are appreciated.<br><br>cheers<br>david<br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <http://lists.koha-community.org/pipermail/koha-devel/attachments/20210422/3aeaa576/attachment-0001.htm><br><br>------------------------------<br><br>Subject: Digest Footer<br><br>_______________________________________________<br>Koha-devel mailing list<br>Koha-devel@lists.koha-community.org<br>https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel<br>website : https://www.koha-community.org/<br>git : https://git.koha-community.org/<br>bugs : https://bugs.koha-community.org/<br><br><br>------------------------------<br><br>End of Koha-devel Digest, Vol 185, Issue 21<br>*******************************************<br></div><br><br>NOTE: You are receiving this mail because, the Requester/Technician wanted you to get notified on this request creation.<br></body></html>