[Koha-devel] Deploying Koha at scale

Wed Aug 19 03:58:09 CEST 2020

Thanks, Agustin.

I haven’t played with Helm yet, but I recognize the Kubernetes manifests in there. It looks interesting. When will these 2 projects be ready for testing?

With koha-docker, have you thought about using “secrets” rather than environmental variables for things like passwords? Reference https://docs.docker.com/compose/compose-file/#secrets. Secrets could be used for the Koha images too following a model like https://github.com/docker-library/mysql/blob/b0f81a33748561ae4e35a09895b2ad112ff89ba6/8.0/docker-entrypoint.sh. That could be useful if someone wanted to use Amazon ECS or something instead of docker-compose too actually.

David Cook

Software Engineer

Prosentient Systems

72/330 Wattle St

Ultimo, NSW 2007

Australia

Office: 02 9212 0899

Online: 02 8005 0595

From: Agustin Moyano <agustinmoyano at theke.io> 
Sent: Wednesday, 19 August 2020 1:51 AM
To: dcook at prosentient.com.au
Cc: Tomás <tomas.cohen at gmail.com>; koha-devel <koha-devel at lists.koha-community.org>
Subject: Re: [Koha-devel] Deploying Koha at scale

Hi David, we've got two separate projects.

One is koha-docker (https://gitlab.com/thekesolutions/koha-docker) that has a "docker-compose" approach, and the second one is koha-helm-chart (https://gitlab.com/thekesolutions/charts/koha-helm-chart) where we will implement the kubernetes cronjob approach.

Please take a look at that one too.

Regards

On Mon, Aug 17, 2020 at 9:51 PM <dcook at prosentient.com.au <mailto:dcook at prosentient.com.au> > wrote:

Hey Tomás,

I remembered your email and I thought that I would check in. How’s work on koha-docker going? It looks interesting.

It’s too bad the “koha-common” package name is already used, as it would be interesting to package just the core libraries and scripts to create a really minimal Koha image to use for the Z39.50, SIP, Indexer, etc. 

I look forward to a time when we don’t need Apache to do CGI scripts and just function as a reverse proxy and front end web server of static assets (or just the a web server of static assets and Traefik or something could be used as the reverse proxy). 

I think that I’m most intrigued with the cron service*, as I think that’s probably the hardest thing to containerize and scale well. 

I’ll have a look at running this up at some point!

*(Kubernetes seems to have a neat approach to cronjobs https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/. Julia Evans talks about it at https://stripe.com/blog/operating-kubernetes. However, that would require a Kubernetes-first design, which makes no sense for Koha.)

David Cook

Software Engineer

Prosentient Systems

72/330 Wattle St

Ultimo, NSW 2007

Australia

Office: 02 9212 0899

Online: 02 8005 0595

From: Koha-devel <koha-devel-bounces at lists.koha-community.org <mailto:koha-devel-bounces at lists.koha-community.org> > On Behalf Of Tomás
Sent: Friday, 12 June 2020 11:57 AM
To: koha-devel <koha-devel at lists.koha-community.org <mailto:koha-devel at lists.koha-community.org> >
Subject: [Koha-devel] Deploying Koha at scale

Hi folks.

Theke Solutions has started a project some time ago, sponsored by ByWater Solutions to generate the tools and pipelines to provide a reliable Docker implementation for deploying Koha in production environments. We have just finished the housekeeping a few days ago so we are not embarrassed by our commit messages and leftovers :-D and want to propose it for community adoption anytime soon.

The project lives here for now: https://gitlab.com/thekesolutions/koha-docker

What you will find there is a Docker image that can be used to run any of the different Koha services/pieces on separate containers, or all inside a single one. We aimed to target different deployment scenarios, from a small library running it on a desktop computer to a multi-cluster K8s setup. A helm chart is part of the roadmap.

You will notice there are a couple issues to be addressed [1]. For instance, the shipped docker-compose.yml is outdated regarding the MySQL/MariaDB server and we definitely need more on the README.

Thanks to ByWater for supporting this effort, and now that people are talking about this, let's start the conversation \o/

[1] https://gitlab.com/thekesolutions/koha-docker/-/issues

El jue., 11 jun. 2020 a las 4:52, <dcook at prosentient.com.au <mailto:dcook at prosentient.com.au> > escribió:

Hi all,

For a long time, I've been thinking about how Koha could scale better, and
I'm wondering what other people are thinking about this topic. I think Mengu
is probably the best expert on this topic that we have on Earth, so
especially curious about his input. 

It's relatively easy to add more Apache web servers, more MySQL nodes, and
more Elasticsearch nodes. but the same might not be said for other parts of
Koha like the following:

1. Zebra
        a. In theory, you could load balance queries to a cluster of Zebra
listeners
        b. But indexing updates to many Zebra databases could be tricky (I
don't know if Zebra's database can support many readers). I suppose you
could alter rebuild_zebra.pl <http://rebuild_zebra.pl>  to update multiple databases in a variety of
different ways. I imagine Mengu could speak to this, as I recall him talking
once about needing to do work to make Zebra scale?

2. I'm not familiar enough with the SIP server to say
        a. Off the top of my head, I think a TCP load balancer would be good
enough here

3. Z3950 Responder? 
        a. I'm not familiar with this daemon

4. What about cronjobs? Only 1 instance should execute the cronjobs.
        a. I suppose a person could disable cronjobs on all Koha nodes
except for 1? It wouldn't be very robust in terms of availability but it
would technically work most of the time. 

Any other thoughts?

David Cook
Systems Librarian
Prosentient Systems
72/330 Wattle St
Ultimo, NSW 2007
Australia

Office: 02 9212 0899
Online: 02 8005 0595

-- 

Tomás Cohen Arazi

Theke Solutions (http://theke.io)
✆ +54 9351 3513384
GPG: B2F3C15F

_______________________________________________
Koha-devel mailing list
Koha-devel at lists.koha-community.org <mailto:Koha-devel at lists.koha-community.org> 
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.koha-community.org/pipermail/koha-devel/attachments/20200819/ff5e042c/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 484 bytes
Desc: not available
URL: <http://lists.koha-community.org/pipermail/koha-devel/attachments/20200819/ff5e042c/attachment-0001.sig>