Re: [Koha-devel] [Koha] Redundant infrastructure for Koha
On 8 November 2016 at 08:50, Michael Kuhn <mik@adminkuhn.ch> wrote:
Hi,
We are looking for a solution to build a redundant infrastructure for Koha.
What we want to do is shown in the diagram http://adminkuhn.ch/download/Redundant.png
The main objective of such a solution is: If one of the Application Servers goes down the other will take over the job without losing any time. If the DB-Server Master goes down we will be able to build the Master out of the Slave and create a new Slave next to the new Master.
Has anyone ever built such an infrastructure with Koha and can give us some advice how to proceed? Is this possible with Koha anyway?
Will shift this to Koha-devel Yes this is possible and quite easy to do. What makes it tricky is Zebra, which you have left out of your diagram. You either need to have the full zebra index on both your application servers (which means modifying rebuild_zebraqueue to make sure it gets built by each machine) Or you need to use something like nfs or another distributed filesystem. Or use rsync etc. It becomes a lot easier with elasticsearch of course, because you just have your elastic cluster and they talk to that. Hope this helps Chris
Best wishes: Michael -- Geschäftsführer · Diplombibliothekar BBS, Informatiker eidg. Fachausweis Admin Kuhn GmbH · Pappelstrasse 20 · 4123 Allschwil · Schweiz T 0041 (0)61 261 55 61 · E mik@adminkuhn.ch · W www.adminkuhn.ch _______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha
Chris, What about the uploaded files? You will need to replicate them to the other server. What about the cross connection of the DB, you will use mysql proxy in the middle? I'm really curious about those answers. Regards. On Mon, Nov 7, 2016 at 5:05 PM, Chris Cormack <chris@bigballofwax.co.nz> wrote:
Hi,
We are looking for a solution to build a redundant infrastructure for Koha.
What we want to do is shown in the diagram http://adminkuhn.ch/download/Redundant.png
The main objective of such a solution is: If one of the Application Servers goes down the other will take over the job without losing any time. If
On 8 November 2016 at 08:50, Michael Kuhn <mik@adminkuhn.ch> wrote: the
DB-Server Master goes down we will be able to build the Master out of the Slave and create a new Slave next to the new Master.
Has anyone ever built such an infrastructure with Koha and can give us some advice how to proceed? Is this possible with Koha anyway?
Will shift this to Koha-devel
Yes this is possible and quite easy to do. What makes it tricky is Zebra, which you have left out of your diagram. You either need to have the full zebra index on both your application servers (which means modifying rebuild_zebraqueue to make sure it gets built by each machine) Or you need to use something like nfs or another distributed filesystem. Or use rsync etc.
It becomes a lot easier with elasticsearch of course, because you just have your elastic cluster and they talk to that.
Hope this helps
Chris
Best wishes: Michael -- Geschäftsführer · Diplombibliothekar BBS, Informatiker eidg. Fachausweis Admin Kuhn GmbH · Pappelstrasse 20 · 4123 Allschwil · Schweiz T 0041 (0)61 261 55 61 · E mik@adminkuhn.ch · W www.adminkuhn.ch _______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
If you allow people to upload files sure. We usually don't that is what a CMS is for. (Uploaded images are stored in the db) And yes talking to a DB is a solved problem, thousands of applications around the world do that already. The only thing that is unique to Koha is Zebra Chris On 8 November 2016 at 09:17, Rodrigo Santellan <rsantellan@gmail.com> wrote:
Chris,
What about the uploaded files? You will need to replicate them to the other server.
What about the cross connection of the DB, you will use mysql proxy in the middle?
I'm really curious about those answers.
Regards.
On Mon, Nov 7, 2016 at 5:05 PM, Chris Cormack <chris@bigballofwax.co.nz> wrote:
On 8 November 2016 at 08:50, Michael Kuhn <mik@adminkuhn.ch> wrote:
Hi,
We are looking for a solution to build a redundant infrastructure for Koha.
What we want to do is shown in the diagram http://adminkuhn.ch/download/Redundant.png
The main objective of such a solution is: If one of the Application Servers goes down the other will take over the job without losing any time. If the DB-Server Master goes down we will be able to build the Master out of the Slave and create a new Slave next to the new Master.
Has anyone ever built such an infrastructure with Koha and can give us some advice how to proceed? Is this possible with Koha anyway?
Will shift this to Koha-devel
Yes this is possible and quite easy to do. What makes it tricky is Zebra, which you have left out of your diagram. You either need to have the full zebra index on both your application servers (which means modifying rebuild_zebraqueue to make sure it gets built by each machine) Or you need to use something like nfs or another distributed filesystem. Or use rsync etc.
It becomes a lot easier with elasticsearch of course, because you just have your elastic cluster and they talk to that.
Hope this helps
Chris
Best wishes: Michael -- Geschäftsführer · Diplombibliothekar BBS, Informatiker eidg. Fachausweis Admin Kuhn GmbH · Pappelstrasse 20 · 4123 Allschwil · Schweiz T 0041 (0)61 261 55 61 · E mik@adminkuhn.ch · W www.adminkuhn.ch _______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
Has anyone tried access Zebra through a network socket instead of the unix one? I was under the impression that that was possible. I've considered a similar setup to what Michael Kuhn is trying build. I already have a database cluster using active-active replication and IP address fail-over with heartbeat. So that's working for me just fine. I've also played with GlusterFS as a shared upload space to serve other applications, so that's also a possibility. It really is Zebra that is the biggest question for me. My hope is that accessing Zebra over the network is possible, then I just dedicate a worker to running and maintaining Zebra, and point the other workers at it. That way worst case is searching doesn't work for the hour or so it would take to spin up Zebra somewhere else if it's server goes down. On Mon, Nov 7, 2016 at 1:19 PM, Chris Cormack <chris@bigballofwax.co.nz> wrote:
If you allow people to upload files sure.
We usually don't that is what a CMS is for. (Uploaded images are stored in the db)
And yes talking to a DB is a solved problem, thousands of applications around the world do that already. The only thing that is unique to Koha is Zebra
Chris
Chris,
What about the uploaded files? You will need to replicate them to the other server.
What about the cross connection of the DB, you will use mysql proxy in
On 8 November 2016 at 09:17, Rodrigo Santellan <rsantellan@gmail.com> wrote: the
middle?
I'm really curious about those answers.
Regards.
On Mon, Nov 7, 2016 at 5:05 PM, Chris Cormack <chris@bigballofwax.co.nz> wrote:
On 8 November 2016 at 08:50, Michael Kuhn <mik@adminkuhn.ch> wrote:
Hi,
We are looking for a solution to build a redundant infrastructure for Koha.
What we want to do is shown in the diagram http://adminkuhn.ch/download/Redundant.png
The main objective of such a solution is: If one of the Application Servers goes down the other will take over the job without losing any time. If the DB-Server Master goes down we will be able to build the Master out of the Slave and create a new Slave next to the new Master.
Has anyone ever built such an infrastructure with Koha and can give us some advice how to proceed? Is this possible with Koha anyway?
Will shift this to Koha-devel
Yes this is possible and quite easy to do. What makes it tricky is Zebra, which you have left out of your diagram. You either need to have the full zebra index on both your application servers (which means modifying rebuild_zebraqueue to make sure it gets built by each machine) Or you need to use something like nfs or another distributed filesystem. Or use rsync etc.
It becomes a lot easier with elasticsearch of course, because you just have your elastic cluster and they talk to that.
Hope this helps
Chris
Best wishes: Michael -- Geschäftsführer · Diplombibliothekar BBS, Informatiker eidg.
Fachausweis
Admin Kuhn GmbH · Pappelstrasse 20 · 4123 Allschwil · Schweiz T 0041 (0)61 261 55 61 · E mik@adminkuhn.ch · W www.adminkuhn.ch _______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha
Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Michael Hafen Washington County School District Technology Department Systems Analyst
Hi, On Mon, Nov 7, 2016 at 4:36 PM, Michael Hafen <michael.hafen@washk12.org> wrote:
Has anyone tried access Zebra through a network socket instead of the unix one? I was under the impression that that was possible.
It is, and it's as easy as changing the following lines in koha-conf.xml from: <listen id="biblioserver" >unix:/var/run/koha/SITE/bibliosocket</listen> <listen id="authorityserver" >unix:/var/run/koha/SITE/authoritysocket</listen> to <listen id="biblioserver" >tcp:HOST_OR_IP:PORT</listen> <listen id="authorityserver" >tcp:HOST_OR_IP:ANOTHER_PORT</listen> Of course, depending on how you arrange things, local tweaks to the indexer jobs would be required to ensure that all of the copies of the Zebra databases got updated. Regards, Galen -- Galen Charlton Infrastructure and Added Services Manager Equinox Software, Inc. / Open Your Library email: gmc@esilibrary.com direct: +1 770-709-5581 cell: +1 404-984-4366 skype: gmcharlt web: http://www.esilibrary.com/ Supporting Koha and Evergreen: http://koha-community.org & http://evergreen-ils.org
Hi, For Zebra and Web-server, this could be a good use case for HA-Proxy. Never used this myself but I have read some documentations about how it can be used to setup redundancy and/or load-balancing between servers for applications that doesn't support this feature out-of the box. For file storage distributed across multiple nodes there is this newcomer called Tahoe-LAFS : https://tahoe-lafs.org/trac/tahoe-lafs Hope this help your purpose. I have to say I've some interests into the question myself, don't hesitate to share your results! Regards, Arthur Le 07/11/2016 à 22:42, Galen Charlton a écrit :
Hi,
On Mon, Nov 7, 2016 at 4:36 PM, Michael Hafen <michael.hafen@washk12.org> wrote:
Has anyone tried access Zebra through a network socket instead of the unix one? I was under the impression that that was possible. It is, and it's as easy as changing the following lines in koha-conf.xml from:
<listen id="biblioserver" >unix:/var/run/koha/SITE/bibliosocket</listen> <listen id="authorityserver" >unix:/var/run/koha/SITE/authoritysocket</listen>
to
<listen id="biblioserver" >tcp:HOST_OR_IP:PORT</listen> <listen id="authorityserver" >tcp:HOST_OR_IP:ANOTHER_PORT</listen>
Of course, depending on how you arrange things, local tweaks to the indexer jobs would be required to ensure that all of the copies of the Zebra databases got updated.
Regards,
Galen
-- Arthur SUZUKI Service informatique des bibliothèques BIBLIOTHÈQUES UNIVERSITAIRES Université Jean Moulin Lyon 3 6 Cours Albert Thomas - B.P. 8242 – 69355 Lyon Cedex 08 ligne directe : +33 (0)4 78 78 79 16 | http://bu.univ-lyon3.fr L'Université Jean Moulin est membre fondateur de l'Université de Lyon
Tahoe-lafs looks interesting if one where locked to cloud storage. I'm planning to deal with storage on the local level, where a simple copy-on-write style system is sufficient. I don't anticipate needing the kind of security provided by Tahoe-lafs, so it seems like overkill to me. As for HA-Proxy, that might be good for client requests to Zebra. The real problem is handling writes to Zebra. Using the queue table in the database would keep one copy up to date, but short of a copy-on-write system I have my doubts about being able to cluster Zebra. Even with a shared file system I expect Zebra tries to lock it's files, which would mean more than one Zebra server on the same directory of the same file system probably wouldn't work. ( I ran into that with Mysql in an earlier attempt at redundancy, and it corrupted many tables when both servers were access the same files at the same time. ) On Tue, Nov 8, 2016 at 2:11 AM, SUZUKI Arthur <arthur.suzuki@univ-lyon3.fr> wrote:
Hi,
For Zebra and Web-server, this could be a good use case for HA-Proxy.
Never used this myself but I have read some documentations about how it can be used to setup redundancy and/or load-balancing between servers for applications that doesn't support this feature out-of the box.
For file storage distributed across multiple nodes there is this newcomer called Tahoe-LAFS :
https://tahoe-lafs.org/trac/tahoe-lafs
Hope this help your purpose.
I have to say I've some interests into the question myself, don't hesitate to share your results!
Regards,
Arthur
Le 07/11/2016 à 22:42, Galen Charlton a écrit :
Hi,
On Mon, Nov 7, 2016 at 4:36 PM, Michael Hafen <michael.hafen@washk12.org> wrote:
Has anyone tried access Zebra through a network socket instead of the unix one? I was under the impression that that was possible.
It is, and it's as easy as changing the following lines in koha-conf.xml from:
<listen id="biblioserver" >unix:/var/run/koha/SITE/bibliosocket</listen> <listen id="authorityserver" >unix:/var/run/koha/SITE/autho ritysocket</listen>
to
<listen id="biblioserver" >tcp:HOST_OR_IP:PORT</listen> <listen id="authorityserver" >tcp:HOST_OR_IP:ANOTHER_PORT</listen>
Of course, depending on how you arrange things, local tweaks to the indexer jobs would be required to ensure that all of the copies of the Zebra databases got updated.
Regards,
Galen
-- Arthur SUZUKI Service informatique des bibliothèques BIBLIOTHÈQUES UNIVERSITAIRES Université Jean Moulin Lyon 3 6 Cours Albert Thomas - B.P. 8242 – 69355 Lyon Cedex 08 ligne directe : +33 (0)4 78 78 79 16 | http://bu.univ-lyon3.fr L'Université Jean Moulin est membre fondateur de l'Université de Lyon
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Michael Hafen Washington County School District Technology Department Systems Analyst
participants (5)
-
Chris Cormack -
Galen Charlton -
Michael Hafen -
Rodrigo Santellan -
SUZUKI Arthur