[Koha-devel] Deploying Koha at scale

[Mengu Yazicioglu]

Hi,

I'd like to share some experiences of our installations and what we have 
done and we haven't.

We've used many servers for apache part, and galera cluster for Mariadb 
part. We used nginx for load balancing and I shared a new entry in the 
wiki about our conf: 
https://wiki.koha-community.org/wiki/Nginx_conf_sample_for_load_balancing_with_many_servers

The main problem always occured from Zebra site as you said David. 
Indexing for more millions records was horrible of course.

After indexing it was easy to copy data and create new server while 
working in a huge VM environment. Not only indexing data but also with 
many requests from users also used a lot of resources of hardwares. 
Since Zebra part was the weak point, we preferred to power database 
servers and change a lot of code just to search directly from database 
instead of using Zebra. This gave us quite big flexibility in our 
installation.

In some problems, if we need to do re-indexing Zebra, we used nigths of 
the day to do that and used some extra servers for indexing and when 
finished we copied the data.

I personally spend a lot of time for Zebra things and I really hate its 
structure. I hope Elastic search will be good for cluster environments.

We also thought Dockerized environment for a scale for large 
installations but we could not find necessary money and time to finish.

Best regards

Mengu

Devinim Ltd.

On 11.06.2020 11:03, Ere Maijala wrote:
> Hi,
>
> This is a welcome discussion, thanks for starting it!
>
> A couple of quick comments:
>
> 1. If you're running Elasticsearch and Z3950 Responder, I can't really
> see a need to run Zebra at all anymore. Is there something that it's
> really needed for?
>
> 3. Z3950 Responder is pretty lightweight, and it'll be the search index
> doing most of the work. But I understand the need from fault tolerance
> point of view. You could use YAZ Proxy or Metaproxy as the load balancer
> for multiple Z3950 Responders. However, that's still a single point
> managing the sessions.
>
> --Ere
>
> dcook at prosentient.com.au kirjoitti 11.6.2020 klo 10.52:
>> Hi all,
>>
>> For a long time, I've been thinking about how Koha could scale better, and
>> I'm wondering what other people are thinking about this topic. I think Mengu
>> is probably the best expert on this topic that we have on Earth, so
>> especially curious about his input.
>>
>> It's relatively easy to add more Apache web servers, more MySQL nodes, and
>> more Elasticsearch nodes. but the same might not be said for other parts of
>> Koha like the following:
>>
>> 1. Zebra
>> 	a. In theory, you could load balance queries to a cluster of Zebra
>> listeners
>> 	b. But indexing updates to many Zebra databases could be tricky (I
>> don't know if Zebra's database can support many readers). I suppose you
>> could alter rebuild_zebra.pl to update multiple databases in a variety of
>> different ways. I imagine Mengu could speak to this, as I recall him talking
>> once about needing to do work to make Zebra scale?
>>
>> 2. I'm not familiar enough with the SIP server to say
>> 	a. Off the top of my head, I think a TCP load balancer would be good
>> enough here
>>
>> 3. Z3950 Responder?
>> 	a. I'm not familiar with this daemon
>>
>> 4. What about cronjobs? Only 1 instance should execute the cronjobs.
>> 	a. I suppose a person could disable cronjobs on all Koha nodes
>> except for 1? It wouldn't be very robust in terms of availability but it
>> would technically work most of the time.
>>
>> Any other thoughts?
>>
>> David Cook
>> Systems Librarian
>> Prosentient Systems
>> 72/330 Wattle St
>> Ultimo, NSW 2007
>> Australia
>>
>> Office: 02 9212 0899
>> Online: 02 8005 0595
>>
>>
>>
>> _______________________________________________
>> Koha-devel mailing list
>> Koha-devel at lists.koha-community.org
>> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
>> website : http://www.koha-community.org/
>> git : http://git.koha-community.org/
>> bugs : http://bugs.koha-community.org/
>>
>
> _______________________________________________
> Koha-devel mailing list
> Koha-devel at lists.koha-community.org
> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
> website : http://www.koha-community.org/
> git : http://git.koha-community.org/
> bugs : http://bugs.koha-community.org/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.koha-community.org/pipermail/koha-devel/attachments/20200611/8c22bf31/attachment.htm>