[Koha-devel] Packaging idea (for decoupling systems, Docker, etc)

Wed Apr 21 08:48:49 CEST 2021

This is another area where I think our thoughts align David.  I've also
wanted to split his into even smaller units for a while..  a clear example
is our move to background jobs. I see those as prime targets for horizontal
scaling.. but as you've highlighted, at the moment to get the library of
core code you may want to write a job you need to install the whole app..
koha-libs would certainly help with that

On Tue, 20 Apr 2021, 2:12 am , <dcook at prosentient.com.au> wrote:

> Hi Mason,
>
> [FYI this started as a response and turned into a
> stream-of-consciousness.][Also really keen to hear from Tomas and Agustin
> about their work with Docker/Kubernetes/Helm/etc.]
>
> The idea with "koha-core" was to separate out the third-party systems away
> from the core application. In this case, "koha-libs" would be about
> separating the core libraries away from the core application.
>
> Consider that the "koha-core" package contains the full Koha application
> (including code, templates, images, other assets, etc) and its application
> dependencies (e.g. at, cron, daemon, starman, sudo, unzip, xmlstarlet, yaz,
> mysql-client). My latest koha-core package alone was 38MB. A "koha-libs"
> package would probably be measured in terms of KB.
>
> I'm thinking about the absolute minimum required to run things like the
> SIP server or some other service/microservice (e.g. z3950-responder, REST
> service, etc). The koha-libs package would need to contain the Koha .pm
> files and koha-libs would depend on koha-perldeps.
>
> In theory, over time, some of the libraries and dependencies would
> probably move out of koha-libs and koha-perldeps to live in the individual
> services, but I don't have a full vision for that yet.
>
> You could think about a koha-libs package as a Perl API rather than a HTTP
> API, which could be used by Koha's existing background services, and it
> could be used by new separate services.
>
> I have a vision for creating a service that depends on koha-libs, points
> at a koha-conf.xml file, and then implements its own functionality separate
> from the main Koha codebase.
>
> It could allow for a lot of power and flexibility.
>
> --
>
> That said, maybe I'm thinking too small and too Debian-focused. Maybe a
> CPAN distribution of Koha/ modules makes more sense (or both koha-libs and
> a CPAN distribution). Apparently we could use the cpanfile to specify the
> dependencies for the CPAN distribution too. Then you could install it on
> any suitable OS/container. In a container environment, you could always
> install the Koha/ modules and dependencies on a volume, and then share that
> volume with your containers. That would have some pros and cons.
>
> I'm mostly just brainstorming. Thinking about how we can re-organize
> things to take advantage of modern technology. But then to what end?
>
> One advantage of Docker is to run different versions of the same software
> on the same virtual machine...
>
> While I was thinking about 1+ containers per Koha library, maybe we do
> just go with a 20.05 Koha container, a 20.11 Koha container, a 21.05 Koha
> container. I suppose each of those versions could have multiple containers
> though. For example, a SIP server container, a Z39.50 responder container,
> a Zebra indexer container, a Zebra server container, etc. Technically
> speaking, if we planned the volumes right, the Koha "etc" config and
> "share" and "var" volumes could all be shared across those containers. Of
> course, that means that all the containers have to be running on the same
> host. At the scale of most Koha libraries, that would probably be fine. I
> suppose the main change here would be to the debian scripts and koha-common
> init script...
>
> --
>
> Overall, I suppose though most vendors would win from improved
> multi-tenancy. Having 100 Starman instances for 50 Koha libraries where
> only 3 libraries might have requests at a time is massive overhead.
>
> I imagine improved multi-tenancy coupled with multiple versions per host
> via Docker... that would be a nicer setup.
>
> Preloading Koha::REST::V1 and getting its work done in the master starman
> process rather than the child processes would be better too...
>
> Ok, that's enough ranting for me for now heh. Too many ideas and never
> enough time.
>
> David Cook
> Software Engineer
> Prosentient Systems
> Suite 7.03
> 6a Glen St
> Milsons Point NSW 2061
> Australia
>
> Office: 02 9212 0899
> Online: 02 8005 0595
>
> -----Original Message-----
> From: Koha-devel <koha-devel-bounces at lists.koha-community.org> On Behalf
> Of Mason James
> Sent: Friday, 16 April 2021 3:50 PM
> To: koha-devel at lists.koha-community.org
> Subject: Re: [Koha-devel] Packaging idea (for decoupling systems, Docker,
> etc)
>
> hi David
>
> i think we may already have this functionality with the 'koha-core'
> package?
>
>
> On 15/04/21 3:32 pm, dcook at prosentient.com.au wrote:
> >
> > Hey all,
> >
> > What do people think about creating a “koha-libs” package which just
> contains Koha’s libraries (ie C4/, Koha/, etc)?
> >
> > I’m not as familiar with DEB packaging as I am with RPM packaging, but I
> recently did it with a RPM-based project I manage. It allowed me to easily
> create other services with the same dependencies. I created these other
> services in other Docker containers, and I was able to use my core
> application libraries without having to install the entire application in
> every container. I share the application configuration file between them
> and that’s it.
> >
> > It’s not a perfect solution. Really what I want is app-service1,
> app-service2, app-service3, and app-common with app-common containing the
> shared libraries. But I like it as an intermediate step.
> >
> > And I would be lying if it didn’t have pros and cons. One con is
> updating a library in app-libs, when I need a change in app-service1, and
> then making sure that all my deployments have the right updated library.
> More overhead than just having a monolithic application. On the other hand,
> if I want to make a change just to app-service1, I can just update it
> without having to affect any other parts of the application. That added
> overhead also gives incentive for writing cleaner testable code which does
> the right thing in the first place.
> >
> > Anyway, just food for thought.
> >
> > David Cook
> >
> > Software Engineer
> >
> > Prosentient Systems
> >
> > Suite 7.03
> >
> > 6a Glen St
> >
> > Milsons Point NSW 2061
> >
> > Australia
> >
> > Office: 02 9212 0899
> >
> > Online: 02 8005 0595
> >
> >
> > _______________________________________________
> > Koha-devel mailing list
> > Koha-devel at lists.koha-community.org
> > https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
> > website : https://www.koha-community.org/ git :
> > https://git.koha-community.org/ bugs :
> > https://bugs.koha-community.org/
>
> _______________________________________________
> Koha-devel mailing list
> Koha-devel at lists.koha-community.org
> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
> website : https://www.koha-community.org/ git :
> https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
>
>
> _______________________________________________
> Koha-devel mailing list
> Koha-devel at lists.koha-community.org
> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
> website : https://www.koha-community.org/
> git : https://git.koha-community.org/
> bugs : https://bugs.koha-community.org/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.koha-community.org/pipermail/koha-devel/attachments/20210421/be7867df/attachment.htm>