<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"><html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body>A new request with request id 18692 has been created by koha-devel-request@lists.koha-community.org. Short info on the request is : <br><br>Title : Koha-devel Digest, Vol 185, Issue 15<br>Category : <br>Description : <div>Send Koha-devel mailing list submissions to<br> koha-devel@lists.koha-community.org<br><br>To subscribe or unsubscribe via the World Wide Web, visit<br> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel<br>or, via email, send a message with subject or body 'help' to<br> koha-devel-request@lists.koha-community.org<br><br>You can reach the person managing the list at<br> koha-devel-owner@lists.koha-community.org<br><br>When replying, please edit your Subject line so it is more specific<br>than "Re: Contents of Koha-devel digest..."<br><br><br>Today's Topics:<br><br> 1. Time to translate: string freeze to prepare April release<br> (Fridolin SOMERS)<br> 2. Re: Packaging idea (for decoupling systems, Docker, etc)<br> (dcook@prosentient.com.au)<br> 3. Optimizing Starman startup (dcook@prosentient.com.au)<br> 4. Re: Optimizing Starman startup (Ere Maijala)<br><br><br>----------------------------------------------------------------------<br><br>Message: 1<br>Date: Mon, 19 Apr 2021 15:06:52 +0200<br>From: Fridolin SOMERS <fridolin.somers@biblibre.com><br>To: koha-translate@lists.koha-community.org, koha-devel<br> <koha-devel@lists.koha-community.org>, "koha@lists.katipo.co.nz"<br> <koha@lists.katipo.co.nz><br>Subject: [Koha-devel] Time to translate: string freeze to prepare<br> April release<br>Message-ID: <0d56c42c-ec7c-1a5e-ce4e-f663877d0912@biblibre.com><br>Content-Type: text/plain; charset=utf-8; format=flowed<br><br>Hi,<br><br>String freeze is into effect.<br>The 20.11.x maintenance branch is preparing for 20.11.05 release.<br>The release is scheduled for around the 26rd.<br><br>This means it's the right time to head over to the translation platform:<br>https://translate.koha-community.org/projects/<br><br>Happy translating 🌎🌍🌏<br><br>-- <br>Fridolin SOMERS <fridolin.somers@biblibre.com><br>Software and system maintainer 🦄<br>BibLibre, France<br><br><br>------------------------------<br><br>Message: 2<br>Date: Tue, 20 Apr 2021 11:12:43 +1000<br>From: <dcook@prosentient.com.au><br>To: "'Mason James'" <mtj@kohaaloha.com>,<br> <koha-devel@lists.koha-community.org><br>Cc: "'Tomas Cohen Arazi'" <tomascohen@theke.io>, "'Agustin Moyano'"<br> <agustinmoyano@theke.io><br>Subject: Re: [Koha-devel] Packaging idea (for decoupling systems,<br> Docker, etc)<br>Message-ID: <012401d73582$44181ad0$cc485070$@prosentient.com.au><br>Content-Type: text/plain; charset="UTF-8"<br><br>Hi Mason,<br><br>[FYI this started as a response and turned into a stream-of-consciousness.][Also really keen to hear from Tomas and Agustin about their work with Docker/Kubernetes/Helm/etc.]<br><br>The idea with "koha-core" was to separate out the third-party systems away from the core application. In this case, "koha-libs" would be about separating the core libraries away from the core application. <br><br>Consider that the "koha-core" package contains the full Koha application (including code, templates, images, other assets, etc) and its application dependencies (e.g. at, cron, daemon, starman, sudo, unzip, xmlstarlet, yaz, mysql-client). My latest koha-core package alone was 38MB. A "koha-libs" package would probably be measured in terms of KB.<br><br>I'm thinking about the absolute minimum required to run things like the SIP server or some other service/microservice (e.g. z3950-responder, REST service, etc). The koha-libs package would need to contain the Koha .pm files and koha-libs would depend on koha-perldeps. <br><br>In theory, over time, some of the libraries and dependencies would probably move out of koha-libs and koha-perldeps to live in the individual services, but I don't have a full vision for that yet. <br><br>You could think about a koha-libs package as a Perl API rather than a HTTP API, which could be used by Koha's existing background services, and it could be used by new separate services.<br><br>I have a vision for creating a service that depends on koha-libs, points at a koha-conf.xml file, and then implements its own functionality separate from the main Koha codebase. <br><br>It could allow for a lot of power and flexibility. <br><br>--<br><br>That said, maybe I'm thinking too small and too Debian-focused. Maybe a CPAN distribution of Koha/ modules makes more sense (or both koha-libs and a CPAN distribution). Apparently we could use the cpanfile to specify the dependencies for the CPAN distribution too. Then you could install it on any suitable OS/container. In a container environment, you could always install the Koha/ modules and dependencies on a volume, and then share that volume with your containers. That would have some pros and cons. <br><br>I'm mostly just brainstorming. Thinking about how we can re-organize things to take advantage of modern technology. But then to what end?<br><br>One advantage of Docker is to run different versions of the same software on the same virtual machine...<br><br>While I was thinking about 1+ containers per Koha library, maybe we do just go with a 20.05 Koha container, a 20.11 Koha container, a 21.05 Koha container. I suppose each of those versions could have multiple containers though. For example, a SIP server container, a Z39.50 responder container, a Zebra indexer container, a Zebra server container, etc. Technically speaking, if we planned the volumes right, the Koha "etc" config and "share" and "var" volumes could all be shared across those containers. Of course, that means that all the containers have to be running on the same host. At the scale of most Koha libraries, that would probably be fine. I suppose the main change here would be to the debian scripts and koha-common init script...<br><br>--<br><br>Overall, I suppose though most vendors would win from improved multi-tenancy. Having 100 Starman instances for 50 Koha libraries where only 3 libraries might have requests at a time is massive overhead. <br><br>I imagine improved multi-tenancy coupled with multiple versions per host via Docker... that would be a nicer setup. <br><br>Preloading Koha::REST::V1 and getting its work done in the master starman process rather than the child processes would be better too...<br><br>Ok, that's enough ranting for me for now heh. Too many ideas and never enough time.<br><br>David Cook<br>Software Engineer<br>Prosentient Systems<br>Suite 7.03<br>6a Glen St<br>Milsons Point NSW 2061<br>Australia<br><br>Office: 02 9212 0899<br>Online: 02 8005 0595<br><br>-----Original Message-----<br>From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of Mason James<br>Sent: Friday, 16 April 2021 3:50 PM<br>To: koha-devel@lists.koha-community.org<br>Subject: Re: [Koha-devel] Packaging idea (for decoupling systems, Docker, etc)<br><br>hi David<br><br>i think we may already have this functionality with the 'koha-core' package?<br><br><br>On 15/04/21 3:32 pm, dcook@prosentient.com.au wrote:<br>><br>> Hey all,<br>><br>> What do people think about creating a “koha-libs” package which just contains Koha’s libraries (ie C4/, Koha/, etc)?<br>><br>> I’m not as familiar with DEB packaging as I am with RPM packaging, but I recently did it with a RPM-based project I manage. It allowed me to easily create other services with the same dependencies. I created these other services in other Docker containers, and I was able to use my core application libraries without having to install the entire application in every container. I share the application configuration file between them and that’s it.<br>><br>> It’s not a perfect solution. Really what I want is app-service1, app-service2, app-service3, and app-common with app-common containing the shared libraries. But I like it as an intermediate step.<br>><br>> And I would be lying if it didn’t have pros and cons. One con is updating a library in app-libs, when I need a change in app-service1, and then making sure that all my deployments have the right updated library. More overhead than just having a monolithic application. On the other hand, if I want to make a change just to app-service1, I can just update it without having to affect any other parts of the application. That added overhead also gives incentive for writing cleaner testable code which does the right thing in the first place.<br>><br>> Anyway, just food for thought.<br>><br>> David Cook<br>><br>> Software Engineer<br>><br>> Prosentient Systems<br>><br>> Suite 7.03<br>><br>> 6a Glen St<br>><br>> Milsons Point NSW 2061<br>><br>> Australia<br>><br>> Office: 02 9212 0899<br>><br>> Online: 02 8005 0595<br>><br>><br>> _______________________________________________<br>> Koha-devel mailing list<br>> Koha-devel@lists.koha-community.org<br>> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel<br>> website : https://www.koha-community.org/ git : <br>> https://git.koha-community.org/ bugs : <br>> https://bugs.koha-community.org/<br><br>_______________________________________________<br>Koha-devel mailing list<br>Koha-devel@lists.koha-community.org<br>https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel<br>website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/<br><br><br><br><br>------------------------------<br><br>Message: 3<br>Date: Tue, 20 Apr 2021 13:15:43 +1000<br>From: <dcook@prosentient.com.au><br>To: <koha-devel@lists.koha-community.org><br>Subject: [Koha-devel] Optimizing Starman startup<br>Message-ID: <013201d73593$72df4b20$589de160$@prosentient.com.au><br>Content-Type: text/plain; charset="utf-8"<br><br>Hi all,<br><br> <br><br>Do you despair when you see the following periodically in "top" when a<br>starman worker is recreated ?<br><br> <br><br> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br><br>9529 my-koha 20 0 460108 197212 17172 R 100.0 0.4 0:03.41<br>/usr/share/koha/api/v1/app.pl<br><br> <br><br>Or the following in top when you install koha-common package or restart the<br>koha-common service?<br><br> <br><br>11101 1-koha 20 0 447232 193320 16076 R 10.6 0.4 0:09.09<br>/usr/share/koha/api/v1/app.pl<br><br>11168 1-koha 20 0 447240 193264 16056 R 10.6 0.4 0:08.72<br>/usr/share/koha/api/v1/app.pl<br><br>11306 2-koha 20 0 447220 193148 16000 R 10.6 0.4 0:08.07<br>/usr/share/koha/api/v1/app.pl<br><br>11543 2-koha 20 0 447232 193036 15828 R 10.6 0.4 0:07.07<br>/usr/share/koha/api/v1/app.pl<br><br>11784 3-koha 20 0 441536 189664 16172 R 10.6 0.4 0:06.04<br>/usr/share/koha/api/v1/app.pl<br><br>11830 3-koha 20 0 439548 187212 15748 R 10.6 0.4 0:05.82<br>/usr/share/koha/api/v1/app.pl<br><br>11831 4-koha 20 0 438620 186344 15748 R 10.6 0.4 0:05.81<br>/usr/share/koha/api/v1/app.pl<br><br>11853 4-koha 20 0 437680 185672 16000 R 10.6 0.4 0:05.79<br>/usr/share/koha/api/v1/app.pl<br><br> <br><br>Well, I still have a lot of investigation left to do, but I notice 1 place<br>that a lot of time taken is here (per worker):<br><br> <br><br> my $validator = JSON::Validator::OpenAPI::Mojolicious->new;<br><br> $validator->load_and_validate_schema(<br><br> $self->home->rel_file("api/v1/swagger/swagger.json"),<br><br> {<br><br> allow_invalid_ref => 1,<br><br> }<br><br> );<br><br> <br><br> push @{$self->routes->namespaces}, 'Koha::Plugin';<br><br> <br><br> my $spec = $validator->schema->data;<br><br> $self->plugin(<br><br> 'Koha::REST::Plugin::PluginRoutes' => {<br><br> spec => $spec,<br><br> validator => $validator<br><br> }<br><br> );<br><br> <br><br> $self->plugin(<br><br> OpenAPI => {<br><br> spec => $spec,<br><br> route => $self->routes->under('/api/v1')->to('Auth#under'),<br><br> allow_invalid_ref =><br><br> 1, # required by our spec because $ref directly under<br><br> # Paths-, Parameters-, Definitions- & Info-object<br><br> # is not allowed by the OpenAPI specification.<br><br> }<br><br>);<br><br> <br><br>Anyone have ideas for improving this? Do we have to validate the schema<br>every time? Can we move the schema validation into a different module and<br>preload it into Starman using the -M flag so that it's done 1 time per<br>Starman master instance rather than 1 time per Starman worker instance?<br><br> <br><br>I find "/usr/share/koha/api/v1/app.pl" to be the bane of deployments, as it<br>puts a massive load on a server, when you have multiple Koha instances on<br>the server. <br><br> <br><br>David Cook<br><br>Software Engineer<br><br>Prosentient Systems<br><br>Suite 7.03<br><br>6a Glen St<br><br>Milsons Point NSW 2061<br><br>Australia<br><br> <br><br>Office: 02 9212 0899<br><br>Online: 02 8005 0595<br><br> <br><br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <http://lists.koha-community.org/pipermail/koha-devel/attachments/20210420/c67b2e3b/attachment-0001.htm><br><br>------------------------------<br><br>Message: 4<br>Date: Tue, 20 Apr 2021 09:47:35 +0300<br>From: Ere Maijala <ere.maijala@helsinki.fi><br>To: koha-devel@lists.koha-community.org<br>Subject: Re: [Koha-devel] Optimizing Starman startup<br>Message-ID: <1ffed076-2760-f6b9-6ce2-d24bf0a48c84@helsinki.fi><br>Content-Type: text/plain; charset=windows-1252; format=flowed<br><br>Hi,<br><br>I did some work on improving it here:<br><br>https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24700<br><br>That shaved a good bit of time from it, but it's still a heavy <br>operation, and it would make sense to<br><br>1.) avoid doing it too often<br><br>2.) cache the results and avoid doing it if results are cached<br><br>If you could address the first one, that'd go a long way. I'm afraid the <br>second one would require changes to the OpenAPI plugin to support caching.<br><br>--Ere<br><br>dcook@prosentient.com.au kirjoitti 20.4.2021 klo 6.15:<br>> Hi all,<br>> <br>> Do you despair when you see the following periodically in “top” when a <br>> starman worker is recreated ?<br>> <br>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br>> <br>> 9529 my-koha 20 0 460108 197212 17172 R 100.0 0.4 0:03.41 <br>> /usr/share/koha/api/v1/app.pl<br>> <br>> Or the following in top when you install koha-common package or restart <br>> the koha-common service?<br>> <br>> 11101 1-koha 20 0 447232 193320 16076 R 10.6 0.4 0:09.09 <br>> /usr/share/koha/api/v1/app.pl<br>> <br>> 11168 1-koha 20 0 447240 193264 16056 R 10.6 0.4 0:08.72 <br>> /usr/share/koha/api/v1/app.pl<br>> <br>> 11306 2-koha 20 0 447220 193148 16000 R 10.6 0.4 0:08.07 <br>> /usr/share/koha/api/v1/app.pl<br>> <br>> 11543 2-koha 20 0 447232 193036 15828 R 10.6 0.4 0:07.07 <br>> /usr/share/koha/api/v1/app.pl<br>> <br>> 11784 3-koha 20 0 441536 189664 16172 R 10.6 0.4 0:06.04 <br>> /usr/share/koha/api/v1/app.pl<br>> <br>> 11830 3-koha 20 0 439548 187212 15748 R 10.6 0.4 0:05.82 <br>> /usr/share/koha/api/v1/app.pl<br>> <br>> 11831 4-koha 20 0 438620 186344 15748 R 10.6 0.4 0:05.81 <br>> /usr/share/koha/api/v1/app.pl<br>> <br>> 11853 4-koha 20 0 437680 185672 16000 R 10.6 0.4 0:05.79 <br>> /usr/share/koha/api/v1/app.pl<br>> <br>> Well, I still have a lot of investigation left to do, but I notice 1 <br>> place that a lot of time taken is here (per worker):<br>> <br>> my $validator = JSON::Validator::OpenAPI::Mojolicious->new;<br>> <br>> $validator->load_and_validate_schema(<br>> <br>> $self->home->rel_file("api/v1/swagger/swagger.json"),<br>> <br>> {<br>> <br>> allow_invalid_ref => 1,<br>> <br>> }<br>> <br>> );<br>> <br>> push @{$self->routes->namespaces}, 'Koha::Plugin';<br>> <br>> my $spec = $validator->schema->data;<br>> <br>> $self->plugin(<br>> <br>> 'Koha::REST::Plugin::PluginRoutes' => {<br>> <br>> spec => $spec,<br>> <br>> validator => $validator<br>> <br>> }<br>> <br>> );<br>> <br>> $self->plugin(<br>> <br>> OpenAPI => {<br>> <br>> spec => $spec,<br>> <br>> route => $self->routes->under('/api/v1')->to('Auth#under'),<br>> <br>> allow_invalid_ref =><br>> <br>> 1, # required by our spec because $ref directly under<br>> <br>> # Paths-, Parameters-, Definitions- & Info-object<br>> <br>> # is not allowed by the OpenAPI specification.<br>> <br>> }<br>> <br>> );<br>> <br>> Anyone have ideas for improving this? Do we have to validate the schema <br>> every time? Can we move the schema validation into a different module <br>> and preload it into Starman using the -M flag so that it’s done 1 time <br>> per Starman master instance rather than 1 time per Starman worker instance?<br>> <br>> I find “/usr/share/koha/api/v1/app.pl” to be the bane of deployments, as <br>> it puts a massive load on a server, when you have multiple Koha <br>> instances on the server.<br>> <br>> David Cook<br>> <br>> Software Engineer<br>> <br>> Prosentient Systems<br>> <br>> Suite 7.03<br>> <br>> 6a Glen St<br>> <br>> Milsons Point NSW 2061<br>> <br>> Australia<br>> <br>> Office: 02 9212 0899<br>> <br>> Online: 02 8005 0595<br>> <br>> <br>> _______________________________________________<br>> Koha-devel mailing list<br>> Koha-devel@lists.koha-community.org<br>> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel<br>> website : https://www.koha-community.org/<br>> git : https://git.koha-community.org/<br>> bugs : https://bugs.koha-community.org/<br>> <br><br>-- <br>Ere Maijala<br>Kansalliskirjasto / The National Library of Finland<br><br><br>------------------------------<br><br>Subject: Digest Footer<br><br>_______________________________________________<br>Koha-devel mailing list<br>Koha-devel@lists.koha-community.org<br>https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel<br>website : https://www.koha-community.org/<br>git : https://git.koha-community.org/<br>bugs : https://bugs.koha-community.org/<br><br><br>------------------------------<br><br>End of Koha-devel Digest, Vol 185, Issue 15<br>*******************************************<br></div><br><br>NOTE: You are receiving this mail because, the Requester/Technician wanted you to get notified on this request creation.<br></body></html>