I am making two proposals that will help "tighten" up our QA procedures a bit in order to facilitate clarity and transparency in our patch submission/acceptance workflow. Currently the workflow is described in the wiki here: http://wiki.koha-community.org/wiki/Bug-enhancement-patch_Workflow#Steps 1. I propose that we modify step 5 to read: "The patch is checked and signed-off by the QA team member. Then the bug status is set to Passed QA" This will ensure that we have clarity that the patch was, indeed, touched by a member of the QA team, as well as increasing the accuracy of QA stats in git. 2. I propose that the RM be the QA of last resort. At present the stats show that the RM is doing the majority of the QA'ing. "Last resort" is a condition evoked by all members of the current QA team acknowledging that no one among them has the time, etc. to do QA on a particular patch the RM feels needs to be pushed OR by a bug remaining in the "Signed Off" status beyond a fixed time period of four weeks. This mechanism would address concerns of bug stagnation by allowing ample time for QA members to at least glance at a bug and determine of they do or do not have time to give to that bug as well as having a catch mechanism for bugs falling through w/o notice. Here are some statistics on sign-offs by the currently listed QA team for commits between the 3.6.0 and 3.8.0 tags: NOTE: 100% of our patches have the minimum required 1 sign-off by any community member Total commits: 1086 Sign-offs by QA Team members: Marcel de Rooy - 43 Jonathan Druart - 5 Paul Poulain - 538 Ian Walls - 25 Total commits w/sign-off by a QA team member: 611 OR 56.3% Total commits w/sign-off by RM: 538 OR 49.5% Total commits w/sign-off by non-RM QA member: 73 OR 6.7% These figures reasonably agree with the stats posted here: http://blog.bigballofwax.co.nz/2012/04/24/statistics-for-koha-3-8-0/ You may reproduce them on your local git repo master branch by using variations of the following: git log --grep="^Signed-off-by: <regexp>" --no-merges --oneline 52c666edce42..5c32a9f811d | wc -l Lets have some discussion on the two proposals above, and I'll put them on the agenda for the next IRC meeting to be voted on. Kind Regards, Chris
Hi, On May 21, 2012, at 1:38 PM, Chris Nighswonger wrote:
1. I propose that we modify step 5 to read:
"The patch is checked and signed-off by the QA team member. Then the bug status is set to Passed QA"
This will ensure that we have clarity that the patch was, indeed, touched by a member of the QA team, as well as increasing the accuracy of QA stats in git.
+1 -- this seems like a obvious procedure documentation fix, hopefully one that just more clearly describes existing practice.
2. I propose that the RM be the QA of last resort. At present the stats show that the RM is doing the majority of the QA'ing. "Last resort" is a condition evoked by all members of the current QA team acknowledging that no one among them has the time, etc. to do QA on a particular patch the RM feels needs to be pushed OR by a bug remaining in the "Signed Off" status beyond a fixed time period of four weeks. This mechanism would address concerns of bug stagnation by allowing ample time for QA members to at least glance at a bug and determine of they do or do not have time to give to that bug as well as having a catch mechanism for bugs falling through w/o notice. [snip] Total commits w/sign-off by non-RM QA member: 73 OR 6.7%
These go hand-in-hand. I agree that reducing the QA burden the RM is a good thing, but in order for that to happen, I think one question that needs an answer first is how do we increase the size of the QA team? Or to phrase it more broadly, what can we do to encourage more people to do QA, whether or not the current QA structure is used as is or changed? Regards, Galen -- Galen Charlton Director of Support and Implementation Equinox Software, Inc. / The Open Source Experts email: gmc@esilibrary.com direct: +1 770-709-5581 cell: +1 404-984-4366 skype: gmcharlt web: http://www.esilibrary.com/ Supporting Koha and Evergreen: http://koha-community.org & http://evergreen-ils.org
Le 21/05/2012 19:38, Chris Nighswonger a écrit :
http://wiki.koha-community.org/wiki/Bug-enhancement-patch_Workflow#Steps
1. I propose that we modify step 5 to read:
"The patch is checked and signed-off by the QA team member. Then the bug status is set to Passed QA" Something I made during the 3.8 release was to add many things to the coding guidelines. My preference goes to QA rules that are clearly defined and explained. That will help QAing a lot, and "anyone" with a good Koha experience, and some time to dedicate should/could do it.
I agree that we *must* have a functional AND a technical review of every patch, the 2 steps are different.
This will ensure that we have clarity that the patch was, indeed, touched by a member of the QA team, as well as increasing the accuracy of QA stats in git. Most QA is done in bugzilla only: when a patch is QAed, it's not signed-off & git bz attach most of the time. (that's also why your numbers below are meaningless : Ian has not made only 25 QA or joubu 5 ! OTOH, when I, as RM, push a patch, I always add my signature, that can be as RM or QA)
2. I propose that the RM be the QA of last resort. At present the stats show that the RM is doing the majority of the QA'ing. As I just wrote, I don't do the majority of QA, (even if I agree I do a lot)
As I've said previously, as RM, I dedicate more than half of my time to this task. I think that we could have someone dedicated full time to QA and someone dedicated full time to sign-off. And until we won't... we will face this kind of trouble. Our workflow is good, but require a large effort we collectively fail to "pay" until now. [ off-topic: BibLibre dedicate a lot of resources to Koha (see statistics on chris_c blog. A lot being "self-sponsored") and can't dedicate more. I think everybody should ask himself seriously "What did I do for Koha last week, what will I do next week ?" ]
"Last resort" is a condition evoked by all members of the current QA team acknowledging that no one among them has the time, etc. to do QA on a particular patch the RM feels needs to be pushed OR by a bug remaining in the "Signed Off" status beyond a fixed time period of four weeks. +1 (and it's already done that way in fact : as member of the QA team, I always order by date when I QA, and start by the oldest patches. I've suggested to change the default order to date, but the idea has not been approved)
-- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc Tel : (33) 4 91 81 35 08
* Paul Poulain (paul.poulain@biblibre.com) wrote:
Le 21/05/2012 19:38, Chris Nighswonger a écrit :
http://wiki.koha-community.org/wiki/Bug-enhancement-patch_Workflow#Steps
1. I propose that we modify step 5 to read:
"The patch is checked and signed-off by the QA team member. Then the bug status is set to Passed QA" Something I made during the 3.8 release was to add many things to the coding guidelines. My preference goes to QA rules that are clearly defined and explained. That will help QAing a lot, and "anyone" with a good Koha experience, and some time to dedicate should/could do it.
I agree that we *must* have a functional AND a technical review of every patch, the 2 steps are different.
This will ensure that we have clarity that the patch was, indeed, touched by a member of the QA team, as well as increasing the accuracy of QA stats in git. Most QA is done in bugzilla only: when a patch is QAed, it's not signed-off & git bz attach most of the time. (that's also why your numbers below are meaningless : Ian has not made only 25 QA or joubu 5 ! OTOH, when I, as RM, push a patch, I always add my signature, that can be as RM or QA)
That's what Chris N was asking for, that a sign off is added when the patch is tested, it must have been applied and tested, so why not sign off and attach it back on the bug at the same time as changing the bug status.
2. I propose that the RM be the QA of last resort. At present the stats show that the RM is doing the majority of the QA'ing. As I just wrote, I don't do the majority of QA, (even if I agree I do a lot)
As I've said previously, as RM, I dedicate more than half of my time to this task. I think that we could have someone dedicated full time to QA and someone dedicated full time to sign-off. And until we won't... we will face this kind of trouble. Our workflow is good, but require a large effort we collectively fail to "pay" until now.
[ off-topic: BibLibre dedicate a lot of resources to Koha (see statistics on chris_c blog. A lot being "self-sponsored") and can't dedicate more. I think everybody should ask himself seriously "What did I do for Koha last week, what will I do next week ?" ]
I'm not sure we want to get into a who does more for Koha contest, nor do I think that was the point of this email. I do think there are too many patches missing getting 2 independent sign offs, and that is what we need to fix. Chris -- Chris Cormack Catalyst IT Ltd. +64 4 803 2238 PO Box 11-053, Manners St, Wellington 6142, New Zealand
Just a quick response on first glance: The numbers in this discussion are imo far from accurate. As for me, I mostly QA without signing off, just to spare time. In most cases I did apply and test (at least some functionality, in other cases much more). I would not recommend doing QA in Bugzilla only, perhaps with the exception of very small things. Adding the rule that QA must do an additional signoff too will put more work (better testing etc.) on the QA team, and will result in more delay. If a QA member does not trust a patch, why not ask for a second signoff (older idea..)? ________________________________________ Van: koha-devel-bounces@lists.koha-community.org [koha-devel-bounces@lists.koha-community.org] namens Chris Cormack [chrisc@catalyst.net.nz] Verzonden: dinsdag 22 mei 2012 10:51 To: Paul Poulain Cc: koha-devel@lists.koha-community.org Onderwerp: Re: [Koha-devel] Proposed QA Enhancements * Paul Poulain (paul.poulain@biblibre.com) wrote:
Le 21/05/2012 19:38, Chris Nighswonger a écrit :
http://wiki.koha-community.org/wiki/Bug-enhancement-patch_Workflow#Steps
1. I propose that we modify step 5 to read:
"The patch is checked and signed-off by the QA team member. Then the bug status is set to Passed QA" Something I made during the 3.8 release was to add many things to the coding guidelines. My preference goes to QA rules that are clearly defined and explained. That will help QAing a lot, and "anyone" with a good Koha experience, and some time to dedicate should/could do it.
I agree that we *must* have a functional AND a technical review of every patch, the 2 steps are different.
This will ensure that we have clarity that the patch was, indeed, touched by a member of the QA team, as well as increasing the accuracy of QA stats in git. Most QA is done in bugzilla only: when a patch is QAed, it's not signed-off & git bz attach most of the time. (that's also why your numbers below are meaningless : Ian has not made only 25 QA or joubu 5 ! OTOH, when I, as RM, push a patch, I always add my signature, that can be as RM or QA)
That's what Chris N was asking for, that a sign off is added when the patch is tested, it must have been applied and tested, so why not sign off and attach it back on the bug at the same time as changing the bug status.
2. I propose that the RM be the QA of last resort. At present the stats show that the RM is doing the majority of the QA'ing. As I just wrote, I don't do the majority of QA, (even if I agree I do a lot)
As I've said previously, as RM, I dedicate more than half of my time to this task. I think that we could have someone dedicated full time to QA and someone dedicated full time to sign-off. And until we won't... we will face this kind of trouble. Our workflow is good, but require a large effort we collectively fail to "pay" until now.
[ off-topic: BibLibre dedicate a lot of resources to Koha (see statistics on chris_c blog. A lot being "self-sponsored") and can't dedicate more. I think everybody should ask himself seriously "What did I do for Koha last week, what will I do next week ?" ]
I'm not sure we want to get into a who does more for Koha contest, nor do I think that was the point of this email. I do think there are too many patches missing getting 2 independent sign offs, and that is what we need to fix. Chris -- Chris Cormack Catalyst IT Ltd. +64 4 803 2238 PO Box 11-053, Manners St, Wellington 6142, New Zealand _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
For my part, I tend to review patches textual to see if they make sense as a changeset. If I cannot tell from reading what this patch does, I'll either apply and test, or ask for clarification. Generally, both of those actions take more effort than reading on Bugzilla, and thus take longer. I agree that it would be preferable to have the actual QA signoff on the patch itself when it gets through QA, so that our git signoff stats are more accurate. That does add the additional overhead of pulling down and applying each patch, then amending the commit and firing back up to BZ... not complicated, but additional steps. Hence why I haven't been doing it reliably. Any QA rules we can put into scripts to automatically check, I'm all for, as that can relieve some of the more tedious work of checking spacing and variable name declarations, and let QA look more in-depth about the patch's implementation. I know we've got a few such tests... anyone who is willing to write more goes in my Nice Person book. My new job is starting to ramp up further, and I'm finding myself doing less QA than I expected. I'll try to rectify that, but if the community feels I'm not doing the job as it needs to be done, I do hope folks will speak up and help us find a remedy to the situation. Cheers, -Ian On Tue, May 22, 2012 at 11:02 AM, Marcel de Rooy <M.de.Rooy@rijksmuseum.nl>wrote:
Just a quick response on first glance: The numbers in this discussion are imo far from accurate. As for me, I mostly QA without signing off, just to spare time. In most cases I did apply and test (at least some functionality, in other cases much more). I would not recommend doing QA in Bugzilla only, perhaps with the exception of very small things. Adding the rule that QA must do an additional signoff too will put more work (better testing etc.) on the QA team, and will result in more delay. If a QA member does not trust a patch, why not ask for a second signoff (older idea..)?
________________________________________ Van: koha-devel-bounces@lists.koha-community.org [ koha-devel-bounces@lists.koha-community.org] namens Chris Cormack [ chrisc@catalyst.net.nz] Verzonden: dinsdag 22 mei 2012 10:51 To: Paul Poulain Cc: koha-devel@lists.koha-community.org Onderwerp: Re: [Koha-devel] Proposed QA Enhancements
* Paul Poulain (paul.poulain@biblibre.com) wrote:
Le 21/05/2012 19:38, Chris Nighswonger a écrit :
http://wiki.koha-community.org/wiki/Bug-enhancement-patch_Workflow#Steps
1. I propose that we modify step 5 to read:
"The patch is checked and signed-off by the QA team member. Then the
bug
status is set to Passed QA" Something I made during the 3.8 release was to add many things to the coding guidelines. My preference goes to QA rules that are clearly defined and explained. That will help QAing a lot, and "anyone" with a good Koha experience, and some time to dedicate should/could do it.
I agree that we *must* have a functional AND a technical review of every patch, the 2 steps are different.
This will ensure that we have clarity that the patch was, indeed, touched by a member of the QA team, as well as increasing the accuracy of QA stats in git. Most QA is done in bugzilla only: when a patch is QAed, it's not signed-off & git bz attach most of the time. (that's also why your numbers below are meaningless : Ian has not made only 25 QA or joubu 5 ! OTOH, when I, as RM, push a patch, I always add my signature, that can be as RM or QA)
That's what Chris N was asking for, that a sign off is added when the patch is tested, it must have been applied and tested, so why not sign off and attach it back on the bug at the same time as changing the bug status.
2. I propose that the RM be the QA of last resort. At present the stats show that the RM is doing the majority of the QA'ing. As I just wrote, I don't do the majority of QA, (even if I agree I do a
lot)
As I've said previously, as RM, I dedicate more than half of my time to this task. I think that we could have someone dedicated full time to QA and someone dedicated full time to sign-off. And until we won't... we will face this kind of trouble. Our workflow is good, but require a large effort we collectively fail to "pay" until now.
[ off-topic: BibLibre dedicate a lot of resources to Koha (see statistics on chris_c blog. A lot being "self-sponsored") and can't dedicate more. I think everybody should ask himself seriously "What did I do for Koha last week, what will I do next week ?" ]
I'm not sure we want to get into a who does more for Koha contest, nor do I think that was the point of this email.
I do think there are too many patches missing getting 2 independent sign offs, and that is what we need to fix.
Chris
-- Chris Cormack Catalyst IT Ltd. +64 4 803 2238 PO Box 11-053, Manners St, Wellington 6142, New Zealand _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/ _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
participants (6)
-
Chris Cormack -
Chris Nighswonger -
Galen Charlton -
Ian Walls -
Marcel de Rooy -
Paul Poulain