Workflow improvement suggestions for security releases
Hello devs, As you must know we had two hard months, several very bad bugs hit us. The release team had to coordinate and it was not an easy task. We noticed some flaws in the workflow and I would like to suggest some improvements to discuss. The main problematic issue is to backport patch series to different stable branches. 1. LTS I think it's time to have a Long Term Support release. We noticed that some people are still using very old versions, having a version that is maintained several years could help them. We could backport critical security bugs only. 4 (5?) years would be great. 2. Communication Once the issues have been reported and fixed, I've alerted the first cycle of people around me. Their job was to alert a second cycle. Should we have a list of people we trust? Ask the (general) mailing list who wants to be in the loop? That means adding them to the security group on bugzilla (or at least adding them when the bug has a fix) and CC them when private discussions take place. 3. Synchronisation Release maintainers are spread around the world (and timezones suck). Getting feedback can take time, several days (like: "try", "don't work", "try again", it's 3 days!). Then when you plan to release on Wednesday, and things are only ready on Thursday you need to wait until Monday as part of the world is still enjoying the weekend! I don't have a solution for that, apart from the Monday postpone or... more anticipation. Same problem for the time of the release, I've picked 12 UTC as the "most convenient" slot for a release, but it won't (ofc) fit everybody's needs. My point was that if we communicated enough beforehand (but not publicly) it should not be a problem. Let me know if you have ideas to improve that! 4. Infrastructure improvement We don't have CI/Jenkins for the security repository. We need one! That must be a top priority of the next cycle. We need to help RMaints and make the security release process easier and less stressful. 5. Apply patches We need a script to apply the patches on the different branches, automatically. That's an easy bit to develop and it will help us a lot. 6. More visibility on the status of the patches RM and RMaints must put their progress on the bug report itself. A comment "Will be pushed, RM had a look at this" or "Backported for..." should be added. That must be added to the "Release process" wiki page. Let us know if you have any questions or remarks. Cheers, Jonathan
1. LTS I think it's time to have a Long Term Support release. We noticed that some people are still using very old versions, having a version that is maintained several years could help them. We could backport critical security bugs only. 4 (5?) years would be great.
This has been proposed in the past. The general consensus at the time was that we as a community want to encourage libraries to stay on the latest versions of Koha and not to hang back on older versions. I'm not necessarily opposed, but such a role will require quite a commitment.
2. Communication Once the issues have been reported and fixed, I've alerted the first cycle of people around me. Their job was to alert a second cycle. Should we have a list of people we trust? Ask the (general) mailing list who wants to be in the loop? That means adding them to the security group on bugzilla (or at least adding them when the bug has a fix) and CC them when private discussions take place.
Sounds good!
3. Synchronisation Release maintainers are spread around the world (and timezones suck). Getting feedback can take time, several days (like: "try", "don't work", "try again", it's 3 days!). Then when you plan to release on Wednesday, and things are only ready on Thursday you need to wait until Monday as part of the world is still enjoying the weekend! I don't have a solution for that, apart from the Monday postpone or... more anticipation. Same problem for the time of the release, I've picked 12 UTC as the "most convenient" slot for a release, but it won't (ofc) fit everybody's needs. My point was that if we communicated enough beforehand (but not publicly) it should not be a problem. Let me know if you have ideas to improve that!
We can already schedule our news posts in advance. Perhaps rmaints should have a non-public area on the ftp server where we could put our tarballs and such, and a cronjob that would automatically move them to the public folder and update the symlinks at 12 UTC every day ( if there are files to move ). I'm not sure how to synchronize the community apt server into this, but I imagine it could be done. If we get into the habit of scheduling everything for release at 12 UTC, security release or not, we can probably get pretty good at it, ironing out the kinks before we have another security release to deal with.
4. Infrastructure improvement We don't have CI/Jenkins for the security repository. We need one! That must be a top priority of the next cycle. We need to help RMaints and make the security release process easier and less stressful.
Agreed!
5. Apply patches We need a script to apply the patches on the different branches, automatically. That's an easy bit to develop and it will help us a lot.
I'm not sure what you mean? Do you mean something that would apply a bug's patch set to say master, stable and oldstable and report on the bug if it doesn't apply to one? Or script in qa-test-tools to do that? Something else entirely?
6. More visibility on the status of the patches RM and RMaints must put their progress on the bug report itself. A comment "Will be pushed, RM had a look at this" or "Backported for..." should be added. That must be added to the "Release process" wiki page.
Sounds good! Kyle
Thanks for your reply, Kyle! (And sorry for my delayed answer, your message hit the spam folder!?) Inline replies below. Le jeu. 7 oct. 2021 à 13:15, Kyle Hall <kyle.m.hall@gmail.com> a écrit :
1. LTS I think it's time to have a Long Term Support release. We noticed that some people are still using very old versions, having a version that is maintained several years could help them. We could backport critical security bugs only. 4 (5?) years would be great.
This has been proposed in the past. The general consensus at the time was that we as a community want to encourage libraries to stay on the latest versions of Koha and not to hang back on older versions. I'm not necessarily opposed, but such a role will require quite a commitment.
For 22.05, Calalyst and Wainui are offering to maintain 19.11, and there is a gap as there is no candidate for 20.05. So it's kind of our first LTS :)
5. Apply patches We need a script to apply the patches on the different branches, automatically. That's an easy bit to develop and it will help us a lot.
I'm not sure what you mean? Do you mean something that would apply a bug's patch set to say master, stable and oldstable and report on the bug if it doesn't apply to one? Or script in qa-test-tools to do that? Something else entirely?
In the first step I was suggesting a script in the release-tools repo to auto-apply a patch set on the different stable branches. But then, our imagination is the limit :)
For 22.05, Calalyst and Wainui are offering to maintain 19.11, and there is a gap as there is no candidate for 20.05. So it's kind of our first LTS :)
I suppose we need to understand Catalyst's upgrade strategy a bit more. It we as a community are going to do LTS, we need to really commit to it. 19.11 will be two years out as of the 21.11 release. Ubuntu LTS releases are every two years. If my math is correct, that would mean that 21.11 would be the next LTS release. Does Catalyst plan on upgrading from 19.11 to 21.11? I'm curious how many libraries would utilize an LTS release and upgrade bi-annually. Do we have any metrics on what is being pulled from the community apt server? If not, is it possible to add tracking? In the first step I was suggesting a script in the release-tools repo
to auto-apply a patch set on the different stable branches. But then, our imagination is the limit :)
That sounds good to me! I can imagine a new bugzilla field for "Fails to apply to" that would be a list of release branches ( 21.05.x, 20.11.x, etc ) so we would be aware. Kyle
We have very few (I think none) 19.11 libraries left. Almost all of our libraries are either running 21.05 or in the process of upgrading). However we have committed to maintain 19.11 for the community for at least one more cycle. It's very low maintenance as its pretty much just security patches from here. If no one else takes over it next cycle we will do it again. Having an LTS release would be useful. By far the majority of Koha libraries are not with a support provider, and if we check hea, there's a large (1000+) number of libraries running a version of 19.11. It would also be one step closer to getting into Debian proper. (a long term dream of mine) Chris On 20 October 2021 7:13:39 am NZDT, Kyle Hall <kyle.m.hall@gmail.com> wrote:
For 22.05, Calalyst and Wainui are offering to maintain 19.11, and there is a gap as there is no candidate for 20.05. So it's kind of our first LTS :)
I suppose we need to understand Catalyst's upgrade strategy a bit more. It we as a community are going to do LTS, we need to really commit to it. 19.11 will be two years out as of the 21.11 release. Ubuntu LTS releases are every two years. If my math is correct, that would mean that 21.11 would be the next LTS release. Does Catalyst plan on upgrading from 19.11 to 21.11? I'm curious how many libraries would utilize an LTS release and upgrade bi-annually. Do we have any metrics on what is being pulled from the community apt server? If not, is it possible to add tracking?
In the first step I was suggesting a script in the release-tools repo
to auto-apply a patch set on the different stable branches. But then, our imagination is the limit :)
That sounds good to me! I can imagine a new bugzilla field for "Fails to apply to" that would be a list of release branches ( 21.05.x, 20.11.x, etc ) so we would be aware.
Kyle
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Thanks Chris! I suppose the next question is, does a 2 year LTS cycle sound right for Koha? Kyle --- http://www.kylehall.info ByWater Solutions ( http://bywatersolutions.com ) Meadville Public Library ( http://www.meadvillelibrary.org ) Crawford County Federated Library System ( http://www.ccfls.org ) On Tue, Oct 19, 2021 at 2:59 PM Chris Cormack <chrisc@catalyst.net.nz> wrote:
We have very few (I think none) 19.11 libraries left. Almost all of our libraries are either running 21.05 or in the process of upgrading). However we have committed to maintain 19.11 for the community for at least one more cycle.
It's very low maintenance as its pretty much just security patches from here. If no one else takes over it next cycle we will do it again.
Having an LTS release would be useful. By far the majority of Koha libraries are not with a support provider, and if we check hea, there's a large (1000+) number of libraries running a version of 19.11.
It would also be one step closer to getting into Debian proper. (a long term dream of mine)
Chris
On 20 October 2021 7:13:39 am NZDT, Kyle Hall <kyle.m.hall@gmail.com> wrote:
For 22.05, Calalyst and Wainui are offering to maintain 19.11, and
there is a gap as there is no candidate for 20.05. So it's kind of our first LTS :)
I suppose we need to understand Catalyst's upgrade strategy a bit more. It we as a community are going to do LTS, we need to really commit to it. 19.11 will be two years out as of the 21.11 release. Ubuntu LTS releases are every two years. If my math is correct, that would mean that 21.11 would be the next LTS release. Does Catalyst plan on upgrading from 19.11 to 21.11? I'm curious how many libraries would utilize an LTS release and upgrade bi-annually. Do we have any metrics on what is being pulled from the community apt server? If not, is it possible to add tracking?
In the first step I was suggesting a script in the release-tools repo
to auto-apply a patch set on the different stable branches. But then, our imagination is the limit :)
That sounds good to me! I can imagine a new bugzilla field for "Fails to apply to" that would be a list of release branches ( 21.05.x, 20.11.x, etc ) so we would be aware.
Kyle
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
participants (3)
-
Chris Cormack -
Jonathan Druart -
Kyle Hall