Deploying Koha at scale
Hi folks. Theke Solutions has started a project some time ago, sponsored by ByWater Solutions to generate the tools and pipelines to provide a reliable Docker implementation for deploying Koha in production environments. We have just finished the housekeeping a few days ago so we are not embarrassed by our commit messages and leftovers :-D and want to propose it for community adoption anytime soon. The project lives here for now: https://gitlab.com/thekesolutions/koha-docker What you will find there is a Docker image that can be used to run any of the different Koha services/pieces on separate containers, or all inside a single one. We aimed to target different deployment scenarios, from a small library running it on a desktop computer to a multi-cluster K8s setup. A helm chart is part of the roadmap. You will notice there are a couple issues to be addressed [1]. For instance, the shipped docker-compose.yml is outdated regarding the MySQL/MariaDB server and we definitely need more on the README. Thanks to ByWater for supporting this effort, and now that people are talking about this, let's start the conversation \o/ [1] https://gitlab.com/thekesolutions/koha-docker/-/issues El jue., 11 jun. 2020 a las 4:52, <dcook@prosentient.com.au> escribió:
Hi all,
For a long time, I've been thinking about how Koha could scale better, and I'm wondering what other people are thinking about this topic. I think Mengu is probably the best expert on this topic that we have on Earth, so especially curious about his input.
It's relatively easy to add more Apache web servers, more MySQL nodes, and more Elasticsearch nodes. but the same might not be said for other parts of Koha like the following:
1. Zebra a. In theory, you could load balance queries to a cluster of Zebra listeners b. But indexing updates to many Zebra databases could be tricky (I don't know if Zebra's database can support many readers). I suppose you could alter rebuild_zebra.pl to update multiple databases in a variety of different ways. I imagine Mengu could speak to this, as I recall him talking once about needing to do work to make Zebra scale?
2. I'm not familiar enough with the SIP server to say a. Off the top of my head, I think a TCP load balancer would be good enough here
3. Z3950 Responder? a. I'm not familiar with this daemon
4. What about cronjobs? Only 1 instance should execute the cronjobs. a. I suppose a person could disable cronjobs on all Koha nodes except for 1? It wouldn't be very robust in terms of availability but it would technically work most of the time.
Any other thoughts?
David Cook Systems Librarian Prosentient Systems 72/330 Wattle St Ultimo, NSW 2007 Australia
Office: 02 9212 0899 Online: 02 8005 0595
-- Tomás Cohen Arazi Theke Solutions (http://theke.io) ✆ +54 9351 3513384 GPG: B2F3C15F
Hey Tomás, I remembered your email and I thought that I would check in. How’s work on koha-docker going? It looks interesting. It’s too bad the “koha-common” package name is already used, as it would be interesting to package just the core libraries and scripts to create a really minimal Koha image to use for the Z39.50, SIP, Indexer, etc. I look forward to a time when we don’t need Apache to do CGI scripts and just function as a reverse proxy and front end web server of static assets (or just the a web server of static assets and Traefik or something could be used as the reverse proxy). I think that I’m most intrigued with the cron service*, as I think that’s probably the hardest thing to containerize and scale well. I’ll have a look at running this up at some point! *(Kubernetes seems to have a neat approach to cronjobs https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/. Julia Evans talks about it at https://stripe.com/blog/operating-kubernetes. However, that would require a Kubernetes-first design, which makes no sense for Koha.) David Cook Software Engineer Prosentient Systems 72/330 Wattle St Ultimo, NSW 2007 Australia Office: 02 9212 0899 Online: 02 8005 0595 From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of Tomás Sent: Friday, 12 June 2020 11:57 AM To: koha-devel <koha-devel@lists.koha-community.org> Subject: [Koha-devel] Deploying Koha at scale Hi folks. Theke Solutions has started a project some time ago, sponsored by ByWater Solutions to generate the tools and pipelines to provide a reliable Docker implementation for deploying Koha in production environments. We have just finished the housekeeping a few days ago so we are not embarrassed by our commit messages and leftovers :-D and want to propose it for community adoption anytime soon. The project lives here for now: https://gitlab.com/thekesolutions/koha-docker What you will find there is a Docker image that can be used to run any of the different Koha services/pieces on separate containers, or all inside a single one. We aimed to target different deployment scenarios, from a small library running it on a desktop computer to a multi-cluster K8s setup. A helm chart is part of the roadmap. You will notice there are a couple issues to be addressed [1]. For instance, the shipped docker-compose.yml is outdated regarding the MySQL/MariaDB server and we definitely need more on the README. Thanks to ByWater for supporting this effort, and now that people are talking about this, let's start the conversation \o/ [1] https://gitlab.com/thekesolutions/koha-docker/-/issues El jue., 11 jun. 2020 a las 4:52, <dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> > escribió: Hi all, For a long time, I've been thinking about how Koha could scale better, and I'm wondering what other people are thinking about this topic. I think Mengu is probably the best expert on this topic that we have on Earth, so especially curious about his input. It's relatively easy to add more Apache web servers, more MySQL nodes, and more Elasticsearch nodes. but the same might not be said for other parts of Koha like the following: 1. Zebra a. In theory, you could load balance queries to a cluster of Zebra listeners b. But indexing updates to many Zebra databases could be tricky (I don't know if Zebra's database can support many readers). I suppose you could alter rebuild_zebra.pl <http://rebuild_zebra.pl> to update multiple databases in a variety of different ways. I imagine Mengu could speak to this, as I recall him talking once about needing to do work to make Zebra scale? 2. I'm not familiar enough with the SIP server to say a. Off the top of my head, I think a TCP load balancer would be good enough here 3. Z3950 Responder? a. I'm not familiar with this daemon 4. What about cronjobs? Only 1 instance should execute the cronjobs. a. I suppose a person could disable cronjobs on all Koha nodes except for 1? It wouldn't be very robust in terms of availability but it would technically work most of the time. Any other thoughts? David Cook Systems Librarian Prosentient Systems 72/330 Wattle St Ultimo, NSW 2007 Australia Office: 02 9212 0899 Online: 02 8005 0595 -- Tomás Cohen Arazi Theke Solutions (http://theke.io) ✆ +54 9351 3513384 GPG: B2F3C15F
Hi David, we've got two separate projects. One is koha-docker (https://gitlab.com/thekesolutions/koha-docker) that has a "docker-compose" approach, and the second one is koha-helm-chart ( https://gitlab.com/thekesolutions/charts/koha-helm-chart) where we will implement the kubernetes cronjob approach. Please take a look at that one too. Regards On Mon, Aug 17, 2020 at 9:51 PM <dcook@prosentient.com.au> wrote:
Hey Tomás,
I remembered your email and I thought that I would check in. How’s work on koha-docker going? It looks interesting.
It’s too bad the “koha-common” package name is already used, as it would be interesting to package just the core libraries and scripts to create a really minimal Koha image to use for the Z39.50, SIP, Indexer, etc.
I look forward to a time when we don’t need Apache to do CGI scripts and just function as a reverse proxy and front end web server of static assets (or just the a web server of static assets and Traefik or something could be used as the reverse proxy).
I think that I’m most intrigued with the cron service*, as I think that’s probably the hardest thing to containerize and scale well.
I’ll have a look at running this up at some point!
*(Kubernetes seems to have a neat approach to cronjobs https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/. Julia Evans talks about it at https://stripe.com/blog/operating-kubernetes. However, that would require a Kubernetes-first design, which makes no sense for Koha.)
David Cook
Software Engineer
Prosentient Systems
72/330 Wattle St
Ultimo, NSW 2007
Australia
Office: 02 9212 0899
Online: 02 8005 0595
*From:* Koha-devel <koha-devel-bounces@lists.koha-community.org> *On Behalf Of *Tomás *Sent:* Friday, 12 June 2020 11:57 AM *To:* koha-devel <koha-devel@lists.koha-community.org> *Subject:* [Koha-devel] Deploying Koha at scale
Hi folks.
Theke Solutions has started a project some time ago, sponsored by ByWater Solutions to generate the tools and pipelines to provide a reliable Docker implementation for deploying Koha in production environments. We have just finished the housekeeping a few days ago so we are not embarrassed by our commit messages and leftovers :-D and want to propose it for community adoption anytime soon.
The project lives here for now: https://gitlab.com/thekesolutions/koha-docker
What you will find there is a Docker image that can be used to run any of the different Koha services/pieces on separate containers, or all inside a single one. We aimed to target different deployment scenarios, from a small library running it on a desktop computer to a multi-cluster K8s setup. A helm chart is part of the roadmap.
You will notice there are a couple issues to be addressed [1]. For instance, the shipped docker-compose.yml is outdated regarding the MySQL/MariaDB server and we definitely need more on the README.
Thanks to ByWater for supporting this effort, and now that people are talking about this, let's start the conversation \o/
[1] https://gitlab.com/thekesolutions/koha-docker/-/issues
El jue., 11 jun. 2020 a las 4:52, <dcook@prosentient.com.au> escribió:
Hi all,
For a long time, I've been thinking about how Koha could scale better, and I'm wondering what other people are thinking about this topic. I think Mengu is probably the best expert on this topic that we have on Earth, so especially curious about his input.
It's relatively easy to add more Apache web servers, more MySQL nodes, and more Elasticsearch nodes. but the same might not be said for other parts of Koha like the following:
1. Zebra a. In theory, you could load balance queries to a cluster of Zebra listeners b. But indexing updates to many Zebra databases could be tricky (I don't know if Zebra's database can support many readers). I suppose you could alter rebuild_zebra.pl to update multiple databases in a variety of different ways. I imagine Mengu could speak to this, as I recall him talking once about needing to do work to make Zebra scale?
2. I'm not familiar enough with the SIP server to say a. Off the top of my head, I think a TCP load balancer would be good enough here
3. Z3950 Responder? a. I'm not familiar with this daemon
4. What about cronjobs? Only 1 instance should execute the cronjobs. a. I suppose a person could disable cronjobs on all Koha nodes except for 1? It wouldn't be very robust in terms of availability but it would technically work most of the time.
Any other thoughts?
David Cook Systems Librarian Prosentient Systems 72/330 Wattle St Ultimo, NSW 2007 Australia
Office: 02 9212 0899 Online: 02 8005 0595
--
Tomás Cohen Arazi
Theke Solutions (http://theke.io) ✆ +54 9351 3513384 GPG: B2F3C15F _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
Thanks, Agustin. I haven’t played with Helm yet, but I recognize the Kubernetes manifests in there. It looks interesting. When will these 2 projects be ready for testing? With koha-docker, have you thought about using “secrets” rather than environmental variables for things like passwords? Reference https://docs.docker.com/compose/compose-file/#secrets. Secrets could be used for the Koha images too following a model like https://github.com/docker-library/mysql/blob/b0f81a33748561ae4e35a09895b2ad1.... That could be useful if someone wanted to use Amazon ECS or something instead of docker-compose too actually. David Cook Software Engineer Prosentient Systems 72/330 Wattle St Ultimo, NSW 2007 Australia Office: 02 9212 0899 Online: 02 8005 0595 From: Agustin Moyano <agustinmoyano@theke.io> Sent: Wednesday, 19 August 2020 1:51 AM To: dcook@prosentient.com.au Cc: Tomás <tomas.cohen@gmail.com>; koha-devel <koha-devel@lists.koha-community.org> Subject: Re: [Koha-devel] Deploying Koha at scale Hi David, we've got two separate projects. One is koha-docker (https://gitlab.com/thekesolutions/koha-docker) that has a "docker-compose" approach, and the second one is koha-helm-chart (https://gitlab.com/thekesolutions/charts/koha-helm-chart) where we will implement the kubernetes cronjob approach. Please take a look at that one too. Regards On Mon, Aug 17, 2020 at 9:51 PM <dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> > wrote: Hey Tomás, I remembered your email and I thought that I would check in. How’s work on koha-docker going? It looks interesting. It’s too bad the “koha-common” package name is already used, as it would be interesting to package just the core libraries and scripts to create a really minimal Koha image to use for the Z39.50, SIP, Indexer, etc. I look forward to a time when we don’t need Apache to do CGI scripts and just function as a reverse proxy and front end web server of static assets (or just the a web server of static assets and Traefik or something could be used as the reverse proxy). I think that I’m most intrigued with the cron service*, as I think that’s probably the hardest thing to containerize and scale well. I’ll have a look at running this up at some point! *(Kubernetes seems to have a neat approach to cronjobs https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/. Julia Evans talks about it at https://stripe.com/blog/operating-kubernetes. However, that would require a Kubernetes-first design, which makes no sense for Koha.) David Cook Software Engineer Prosentient Systems 72/330 Wattle St Ultimo, NSW 2007 Australia Office: 02 9212 0899 Online: 02 8005 0595 From: Koha-devel <koha-devel-bounces@lists.koha-community.org <mailto:koha-devel-bounces@lists.koha-community.org> > On Behalf Of Tomás Sent: Friday, 12 June 2020 11:57 AM To: koha-devel <koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org> > Subject: [Koha-devel] Deploying Koha at scale Hi folks. Theke Solutions has started a project some time ago, sponsored by ByWater Solutions to generate the tools and pipelines to provide a reliable Docker implementation for deploying Koha in production environments. We have just finished the housekeeping a few days ago so we are not embarrassed by our commit messages and leftovers :-D and want to propose it for community adoption anytime soon. The project lives here for now: https://gitlab.com/thekesolutions/koha-docker What you will find there is a Docker image that can be used to run any of the different Koha services/pieces on separate containers, or all inside a single one. We aimed to target different deployment scenarios, from a small library running it on a desktop computer to a multi-cluster K8s setup. A helm chart is part of the roadmap. You will notice there are a couple issues to be addressed [1]. For instance, the shipped docker-compose.yml is outdated regarding the MySQL/MariaDB server and we definitely need more on the README. Thanks to ByWater for supporting this effort, and now that people are talking about this, let's start the conversation \o/ [1] https://gitlab.com/thekesolutions/koha-docker/-/issues El jue., 11 jun. 2020 a las 4:52, <dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> > escribió: Hi all, For a long time, I've been thinking about how Koha could scale better, and I'm wondering what other people are thinking about this topic. I think Mengu is probably the best expert on this topic that we have on Earth, so especially curious about his input. It's relatively easy to add more Apache web servers, more MySQL nodes, and more Elasticsearch nodes. but the same might not be said for other parts of Koha like the following: 1. Zebra a. In theory, you could load balance queries to a cluster of Zebra listeners b. But indexing updates to many Zebra databases could be tricky (I don't know if Zebra's database can support many readers). I suppose you could alter rebuild_zebra.pl <http://rebuild_zebra.pl> to update multiple databases in a variety of different ways. I imagine Mengu could speak to this, as I recall him talking once about needing to do work to make Zebra scale? 2. I'm not familiar enough with the SIP server to say a. Off the top of my head, I think a TCP load balancer would be good enough here 3. Z3950 Responder? a. I'm not familiar with this daemon 4. What about cronjobs? Only 1 instance should execute the cronjobs. a. I suppose a person could disable cronjobs on all Koha nodes except for 1? It wouldn't be very robust in terms of availability but it would technically work most of the time. Any other thoughts? David Cook Systems Librarian Prosentient Systems 72/330 Wattle St Ultimo, NSW 2007 Australia Office: 02 9212 0899 Online: 02 8005 0595 -- Tomás Cohen Arazi Theke Solutions (http://theke.io) ✆ +54 9351 3513384 GPG: B2F3C15F _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
participants (4)
-
Agustin Moyano -
dcook＠prosentient.com.au -
Marcel de Rooy -
Tomás