This is another area where I think our thoughts align David. I've also wanted to split his into even smaller units for a while.. a clear example is our move to background jobs. I see those as prime targets for horizontal scaling.. but as you've highlighted, at the moment to get the library of core code you may want to write a job you need to install the whole app.. koha-libs would certainly help with that On Tue, 20 Apr 2021, 2:12 am , <dcook@prosentient.com.au> wrote:
Hi Mason,
[FYI this started as a response and turned into a stream-of-consciousness.][Also really keen to hear from Tomas and Agustin about their work with Docker/Kubernetes/Helm/etc.]
The idea with "koha-core" was to separate out the third-party systems away from the core application. In this case, "koha-libs" would be about separating the core libraries away from the core application.
Consider that the "koha-core" package contains the full Koha application (including code, templates, images, other assets, etc) and its application dependencies (e.g. at, cron, daemon, starman, sudo, unzip, xmlstarlet, yaz, mysql-client). My latest koha-core package alone was 38MB. A "koha-libs" package would probably be measured in terms of KB.
I'm thinking about the absolute minimum required to run things like the SIP server or some other service/microservice (e.g. z3950-responder, REST service, etc). The koha-libs package would need to contain the Koha .pm files and koha-libs would depend on koha-perldeps.
In theory, over time, some of the libraries and dependencies would probably move out of koha-libs and koha-perldeps to live in the individual services, but I don't have a full vision for that yet.
You could think about a koha-libs package as a Perl API rather than a HTTP API, which could be used by Koha's existing background services, and it could be used by new separate services.
I have a vision for creating a service that depends on koha-libs, points at a koha-conf.xml file, and then implements its own functionality separate from the main Koha codebase.
It could allow for a lot of power and flexibility.
--
That said, maybe I'm thinking too small and too Debian-focused. Maybe a CPAN distribution of Koha/ modules makes more sense (or both koha-libs and a CPAN distribution). Apparently we could use the cpanfile to specify the dependencies for the CPAN distribution too. Then you could install it on any suitable OS/container. In a container environment, you could always install the Koha/ modules and dependencies on a volume, and then share that volume with your containers. That would have some pros and cons.
I'm mostly just brainstorming. Thinking about how we can re-organize things to take advantage of modern technology. But then to what end?
One advantage of Docker is to run different versions of the same software on the same virtual machine...
While I was thinking about 1+ containers per Koha library, maybe we do just go with a 20.05 Koha container, a 20.11 Koha container, a 21.05 Koha container. I suppose each of those versions could have multiple containers though. For example, a SIP server container, a Z39.50 responder container, a Zebra indexer container, a Zebra server container, etc. Technically speaking, if we planned the volumes right, the Koha "etc" config and "share" and "var" volumes could all be shared across those containers. Of course, that means that all the containers have to be running on the same host. At the scale of most Koha libraries, that would probably be fine. I suppose the main change here would be to the debian scripts and koha-common init script...
--
Overall, I suppose though most vendors would win from improved multi-tenancy. Having 100 Starman instances for 50 Koha libraries where only 3 libraries might have requests at a time is massive overhead.
I imagine improved multi-tenancy coupled with multiple versions per host via Docker... that would be a nicer setup.
Preloading Koha::REST::V1 and getting its work done in the master starman process rather than the child processes would be better too...
Ok, that's enough ranting for me for now heh. Too many ideas and never enough time.
David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia
Office: 02 9212 0899 Online: 02 8005 0595
-----Original Message----- From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of Mason James Sent: Friday, 16 April 2021 3:50 PM To: koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Packaging idea (for decoupling systems, Docker, etc)
hi David
i think we may already have this functionality with the 'koha-core' package?
On 15/04/21 3:32 pm, dcook@prosentient.com.au wrote:
Hey all,
What do people think about creating a “koha-libs” package which just
contains Koha’s libraries (ie C4/, Koha/, etc)?
I’m not as familiar with DEB packaging as I am with RPM packaging, but I
recently did it with a RPM-based project I manage. It allowed me to easily create other services with the same dependencies. I created these other services in other Docker containers, and I was able to use my core application libraries without having to install the entire application in every container. I share the application configuration file between them and that’s it.
It’s not a perfect solution. Really what I want is app-service1,
app-service2, app-service3, and app-common with app-common containing the shared libraries. But I like it as an intermediate step.
And I would be lying if it didn’t have pros and cons. One con is
updating a library in app-libs, when I need a change in app-service1, and then making sure that all my deployments have the right updated library. More overhead than just having a monolithic application. On the other hand, if I want to make a change just to app-service1, I can just update it without having to affect any other parts of the application. That added overhead also gives incentive for writing cleaner testable code which does the right thing in the first place.
Anyway, just food for thought.
David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/