Hi, for example if we rename a DB column in Koha now we need to stop all the koha web server instances (or they stop responding because because koha checks that db revision is correct), do DB upgrade, start all Koha web server instances. With rolling upgrade we would first create another column with the same name and write & read data to both columns during the 1st step of upgrade, then in 2nd step when all web server nodes are using latest version we could upgrade again to newer version and just delete the old column and start using new one. Ahaha, there is probably better explanation online if you search "rolling database schema upgrade" but hopefully the above clarfied my intention. Joonas On 02/03/2021 16:47, Magnus Enger wrote:

Den 02.03.2021 15:33, skrev Joonas Kylmälä:

...

Hi,

I opened a bug report for introducing rolling DB upgrade support for Koha. I would like to hear if you have anything to object to that or should we just go ahead and develop that? :)

I must confess I am not 100% certain what you mean by rolling updates here. Could you give some more details?

Best regards, Magnus Libriotech

_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/

-- Joonas Kylmälä Tietojärjestelmäasiantuntija Kansalliskirjasto Kirjastoverkkopalvelut PL 15 (Unioninkatu 36) 00014 Helsingin yliopisto

Le 02/03/2021 à 16:00, Joonas Kylmälä a écrit :

With rolling upgrade we would first create another column with the new name and write & read data to both columns during the 1st step of upgrade, then in 2nd step when all web server nodes are using latest version we could upgrade again to newer version and just delete the old column and start using new one.

Does it mean that the application (Perl) code should be aware of that intermediary state where both columns exist at the same time ? If so, for how long should it be aware of that old database state ? Does it mean that, for every database schema update, you have to do something like this: - git reset --hard commit_for_1st_step - updatedatabase (add new column) && reload starman - git reset --hard commit_for_2nd_step - updatedatabase (remove old column) && reload starman ? Do you know another open source software that allows rolling upgrade ? It seems like a great idea, but from what I just read online it looks really hard to do it correctly. -- Julian Maurice BibLibre

Hi, On 03/03/2021 00:47, dcook@prosentient.com.au wrote:

It seems to me that we would have to train every developer on how to write their Perl code and database changes to work with this conceptual model.

Yeah, that's correct if there is DB changes.

That being said, I am intrigued. I hate downtime as much as the next person.

Downtime, removal of odd maintenance hours and shorter feedback loop are the main benefits I see here.

But how do you propose that the Koha community (and not just your organisation) would make this work? At the moment, the Koha community releases new versions once every month. Some libraries pick those upgrades up automatically, some have sysadmins initiate upgrades, etc. I think we have to assume that no human intervention is a requirement for Koha. So I'm not sure how a 2-step upgrade would work for libraries without sysadmins who were specifically knowledgeable about Koha.

For such installations, non-distributed ones, there would be no change. You could just upgrade the debian package and it would do all the upgrade steps. If you have Koha web server on multiple different servers then you have to do similar coordination as you have had to do already with Koha but just with different commands. Joonas -- Joonas Kylmälä Tietojärjestelmäasiantuntija Kansalliskirjasto Kirjastoverkkopalvelut PL 15 (Unioninkatu 36) 00014 Helsingin yliopisto

Hi On 02/03/2021 17:56, Julian Maurice wrote:

Does it mean that the application (Perl) code should be aware of that intermediary state where both columns exist at the same time ? If so, for how long should it be aware of that old database state ?

Yup. And answer to the second question: as long as the DB migration takes, sometimes just a few seconds, sometimes hours (we had e.g. some character encoding change in biblio metadata table if I'm not mistaken and it took a loong time).

Does it mean that, for every database schema update, you have to do something like this: - git reset --hard commit_for_1st_step - updatedatabase (add new column) && reload starman - git reset --hard commit_for_2nd_step - updatedatabase (remove old column) && reload starman ?

If you just do the reload starman before updatedatabase then yeah, I think this represents the idea :) I found more verbose explanation from here: http://alexandre-masselot.blogspot.com/2012/08/continuous-deployment-in-perl...

Do you know another open source software that allows rolling upgrade ? It seems like a great idea, but from what I just read online it looks really hard to do it correctly.

I don't know any other software but not sure it is that difficult. What I gather, we could use e.g. Jenkins to push the code / new debian package that would be created for every commit. Jenkins would make sure db schema migration is finished before installing a newer version and the steps would be repeated until we are at the latest release. And I want to add to to the other email I sent earlier, to clarify the non-distributed setup for David: the one node setup would probably be best to be done in the current fashion, shutdown koha, run all db migrations, start koha. It seems hacky and useless to try do downtimeless migration with that given you also don't care the service being down during HW maintanance, etc. Joonas -- Joonas Kylmälä Tietojärjestelmäasiantuntija Kansalliskirjasto Kirjastoverkkopalvelut PL 15 (Unioninkatu 36) 00014 Helsingin yliopisto

Hi, On 03/03/2021 11:39, Julian Maurice wrote:

I'm not too much concerned about the "push-test-release" process, but more about how we consistently make the code aware of the multiple possible database states. Surely we'd have to enforce a new set of rules for writing database updates. And we'd have to test these changes very carefully, since a tiny mistake could mean a loss of data. Of course you have backups, but DB rollback means downtime, right ?

Depends on the rollback I guess, some things you could probably revert without downtime. The possibility of loss of data due to programming error happens now as well and is no different.

So, that sounds like a lot of extra work for avoiding a few seconds of downtime (most of the time).

Not really sure it is more work than writing the current db upgrade procedures, maybe tiny bit, and a bit more if you want to be able to revert the DB upgrade as well (which is optional and I don't think we should even consider that for MVP implementation of this).

Also, what about template translations ? It takes some time to generate translated templates, and it can do weird things if the new code is used while using the old templates (or the old code with the new templates). Wouldn't that prevent zero-downtime upgrade ?

You redirect the traffic during upgrade to other nodes that are in working state and after the node being upgraded is ready to receive traffic you then can re-enable the traffic redirection there. Joonas -- Joonas Kylmälä Tietojärjestelmäasiantuntija Kansalliskirjasto Kirjastoverkkopalvelut PL 15 (Unioninkatu 36) 00014 Helsingin yliopisto

On 03/03/2021 12:25, Julian Maurice wrote:

Le 03/03/2021 à 11:18, Joonas Kylmälä a écrit :

...

Not really sure it is more work than writing the current db upgrade procedures, maybe tiny bit, and a bit more if you want to be able to revert the DB upgrade as well (which is optional and I don't think we should even consider that for MVP implementation of this).

You at least need to modify the code so that it handles the intermediary state, don't you ? In the case of a column/table rename, you need to make sure both columns/tables stay in sync until the whole upgrade process is finished.

You are right, that's the extra stuff needed. -- Joonas Kylmälä Tietojärjestelmäasiantuntija Kansalliskirjasto Kirjastoverkkopalvelut PL 15 (Unioninkatu 36) 00014 Helsingin yliopisto