[Koha-devel] Packaging idea (for decoupling systems, Docker, etc)

Wed Apr 21 09:41:47 CEST 2021

I like to think that we’re two peas in a pod. (I don’t know why I use so many English idioms on the listserv. I swear that I don’t actually use them in my every day life.)

I’m not sure how easy it is to split a package up with DEB packages but it’s so easy with RPM packages. I feel like there’s no harm in trying. 

I haven’t been motivated to code outside of work hours for a while now, but maybe I would if I had that koha-libs package. I like the idea of spending free time working on little services that can live in their own Docker containers. 

(Actually, I already have something like that for Shibboleth. You can’t install it on Ubuntu 18.04 due to broken dependencies, so what I do is have a Docker container running Debian and do everything with Shibboleth in there, and I just proxy from the Docker host that runs Koha to the Docker container to do the auth. I then pass back a one-shot auth token to authenticate back into Koha. Works a treat. If I had access to the Koha core libs, I’d be able to do it even more seamlessly…)

David Cook

Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

Office: 02 9212 0899

Online: 02 8005 0595

From: Renvoize, Martin <martin.renvoize at ptfs-europe.com> 
Sent: Wednesday, 21 April 2021 4:49 PM
To: David Cook <dcook at prosentient.com.au>
Cc: Mason James <mtj at kohaaloha.com>; Koha Devel <koha-devel at lists.koha-community.org>; Tomas Cohen Arazi <tomascohen at theke.io>
Subject: Re: [Koha-devel] Packaging idea (for decoupling systems, Docker, etc)

This is another area where I think our thoughts align David.  I've also wanted to split his into even smaller units for a while..  a clear example is our move to background jobs. I see those as prime targets for horizontal scaling.. but as you've highlighted, at the moment to get the library of core code you may want to write a job you need to install the whole app.. koha-libs would certainly help with that

On Tue, 20 Apr 2021, 2:12 am , <dcook at prosentient.com.au <mailto:dcook at prosentient.com.au> > wrote:

Hi Mason,

[FYI this started as a response and turned into a stream-of-consciousness.][Also really keen to hear from Tomas and Agustin about their work with Docker/Kubernetes/Helm/etc.]

The idea with "koha-core" was to separate out the third-party systems away from the core application. In this case, "koha-libs" would be about separating the core libraries away from the core application. 

Consider that the "koha-core" package contains the full Koha application (including code, templates, images, other assets, etc) and its application dependencies (e.g. at, cron, daemon, starman, sudo, unzip, xmlstarlet, yaz, mysql-client). My latest koha-core package alone was 38MB. A "koha-libs" package would probably be measured in terms of KB.

I'm thinking about the absolute minimum required to run things like the SIP server or some other service/microservice (e.g. z3950-responder, REST service, etc). The koha-libs package would need to contain the Koha .pm files and koha-libs would depend on koha-perldeps. 

In theory, over time, some of the libraries and dependencies would probably move out of koha-libs and koha-perldeps to live in the individual services, but I don't have a full vision for that yet.  

You could think about a koha-libs package as a Perl API rather than a HTTP API, which could be used by Koha's existing background services, and it could be used by new separate services.

I have a vision for creating a service that depends on koha-libs, points at a koha-conf.xml file, and then implements its own functionality separate from the main Koha codebase. 

It could allow for a lot of power and flexibility. 

--

That said, maybe I'm thinking too small and too Debian-focused. Maybe a CPAN distribution of Koha/ modules makes more sense (or both koha-libs and a CPAN distribution). Apparently we could use the cpanfile to specify the dependencies for the CPAN distribution too. Then you could install it on any suitable OS/container. In a container environment, you could always install the Koha/ modules and dependencies on a volume, and then share that volume with your containers. That would have some pros and cons. 

I'm mostly just brainstorming. Thinking about how we can re-organize things to take advantage of modern technology. But then to what end?

One advantage of Docker is to run different versions of the same software on the same virtual machine...

While I was thinking about 1+ containers per Koha library, maybe we do just go with a 20.05 Koha container, a 20.11 Koha container, a 21.05 Koha container. I suppose each of those versions could have multiple containers though. For example, a SIP server container, a Z39.50 responder container, a Zebra indexer container, a Zebra server container, etc. Technically speaking, if we planned the volumes right, the Koha "etc" config and "share" and "var" volumes could all be shared across those containers. Of course, that means that all the containers have to be running on the same host. At the scale of most Koha libraries, that would probably be fine. I suppose the main change here would be to the debian scripts and koha-common init script...

--

Overall, I suppose though most vendors would win from improved multi-tenancy. Having 100 Starman instances for 50 Koha libraries where only 3 libraries might have requests at a time is massive overhead. 

I imagine improved multi-tenancy coupled with multiple versions per host via Docker... that would be a nicer setup. 

Preloading Koha::REST::V1 and getting its work done in the master starman process rather than the child processes would be better too...

Ok, that's enough ranting for me for now heh. Too many ideas and never enough time.

David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia

Office: 02 9212 0899
Online: 02 8005 0595

-----Original Message-----
From: Koha-devel <koha-devel-bounces at lists.koha-community.org <mailto:koha-devel-bounces at lists.koha-community.org> > On Behalf Of Mason James
Sent: Friday, 16 April 2021 3:50 PM
To: koha-devel at lists.koha-community.org <mailto:koha-devel at lists.koha-community.org> 
Subject: Re: [Koha-devel] Packaging idea (for decoupling systems, Docker, etc)

hi David

i think we may already have this functionality with the 'koha-core' package?

On 15/04/21 3:32 pm, dcook at prosentient.com.au <mailto:dcook at prosentient.com.au>  wrote:
>
> Hey all,
>
> What do people think about creating a “koha-libs” package which just contains Koha’s libraries (ie C4/, Koha/, etc)?
>
> I’m not as familiar with DEB packaging as I am with RPM packaging, but I recently did it with a RPM-based project I manage. It allowed me to easily create other services with the same dependencies. I created these other services in other Docker containers, and I was able to use my core application libraries without having to install the entire application in every container. I share the application configuration file between them and that’s it.
>
> It’s not a perfect solution. Really what I want is app-service1, app-service2, app-service3, and app-common with app-common containing the shared libraries. But I like it as an intermediate step.
>
> And I would be lying if it didn’t have pros and cons. One con is updating a library in app-libs, when I need a change in app-service1, and then making sure that all my deployments have the right updated library. More overhead than just having a monolithic application. On the other hand, if I want to make a change just to app-service1, I can just update it without having to affect any other parts of the application. That added overhead also gives incentive for writing cleaner testable code which does the right thing in the first place.
>
> Anyway, just food for thought.
>
> David Cook
>
> Software Engineer
>
> Prosentient Systems
>
> Suite 7.03
>
> 6a Glen St
>
> Milsons Point NSW 2061
>
> Australia
>
> Office: 02 9212 0899
>
> Online: 02 8005 0595
>
>
> _______________________________________________
> Koha-devel mailing list
> Koha-devel at lists.koha-community.org <mailto:Koha-devel at lists.koha-community.org> 
> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
> website : https://www.koha-community.org/ git : 
> https://git.koha-community.org/ bugs : 
> https://bugs.koha-community.org/

_______________________________________________
Koha-devel mailing list
Koha-devel at lists.koha-community.org <mailto:Koha-devel at lists.koha-community.org> 
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/

_______________________________________________
Koha-devel mailing list
Koha-devel at lists.koha-community.org <mailto:Koha-devel at lists.koha-community.org> 
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.koha-community.org/pipermail/koha-devel/attachments/20210421/0b0f4c9e/attachment.htm>