Hi koha-devel, On the last Koha team meeting of March 20th, 2006, we discussed about the position INEO (me for instance) could officialy take on the project. Joshua and Paul proposed me to be Quality Assurance (QA) manager, and I personnaly accepted this role for a two months try. I've just asked my INEO managers for a formal validation. On IRC, the definition about QA manager was discussed, because everybody doesn't define it the same way. Paul said "responsible for heavy testing" and I answered I did not agree with this point of view, not in OSS model. thd asked what difference I made between open and closed source QA. Let's answer this question by mail, as requested by Russ. See details on IRC logs [1]. I first answer to the question "what difference I make between open and closed source QA", then I'll describe what is my vision of QA manager for an OSS project like Koha. QA, proprietay Vs OSS ===================== I've worked on the two sides of softwares : proprietary and OSS. I think I understand the advantages and drawbacks of each model, concerning QA. In a proprietary model, the software *must* be /stable/ when reaching customers. In my vision, /stable/ means: - without bug that may corrupt customers data - without blocking bug Consequently, a proprietary software vendor needs to have a QA team. This team assures the stability of releases. No right to mistakes because financial penalties will be requested by customers. OSS model is different because it *involves* customer on QA. This is a very important point in my opinion. An OSS project must provide efficient tools to communicate with core dev team. Customers/users are willing to propose patches. OSS model partially moves QA to customers. Not completely of course, and I'll describe muy vision of QA management later. I'm sure Koha developers have all read Eric Raymond essay "The Cathedral & the Bazaar" [2] (if not, I encourage the reading). In this essay, you can read the summarize of Linus's law : "Given enough eyeballs, all bugs are shallow.". Koha team will never have enough eyeballs, this is why customers must be involved in bug tracking, from notification to test & validation. Having worked nearly 3 years in a proprietary software company, I can assure you that QA is one of the most critical task, and very often underestimated since it costs a lot while not giving back money directly. QA team needs to grow at the same speed as development team output, and even more because complexity increases more than linearly. OSS offers a solution to assure software quality without costing a lot more. Indeed, the more customers, the more requested features, the more tests required. If customers are involved in QA process, the number of testers will increase the same as the number of features. My conclusion is that in OSS, QA dedicated team don't need to be as big as development team. My explanations are very theoretical, in the reality OSS vendors have to "release often, release early" [2] and involve their customer at an early stage of deployment, not only before production deployment. This is why HEAD builds, 3.0-RCx are necessary. I think some customers will accept to play the role of beta tester. I ask Paul, Joshua, Russ and other Koha services vendor to give their opinion. My QA manager vision on Koha ============================ I come to my vision of QA manager on Koha project. The most important of the QA manager is to know the content of the bug tracker: 1. being notified on every update 2. dispatch issue to developers (with developer authorization) 3. check that each issue is assigned/treated 4. reduce time between bug notification and its closure It is important that no bug remain a too long time unassigned/untreated. If this happens, customer won't feel useful to involve herself. A bug rapidly corrected with a efficient communication between reporter and developer will encourage customers to report bugs, thus improving Koha quality in the long run. An additionnal task of the QA manager is Bug Squashing Party (BSP) coordination. I need to be explained how Koha worked on this in the past. I'm waiting for your feedback. [1] http://tinyurl.com/n4234 [2] http://www.catb.org/~esr/writings/cathedral-bazaar/cathedral-bazaar/ Cheers, -- Pierrick LE GALL INEO media system
Hi Pierrick, Sorry so late in getting back to you on this ... here we go: On Tue, Mar 21, 2006 at 02:32:51PM +0100, Pierrick LE GALL wrote:
On the last Koha team meeting of March 20th, 2006, we discussed about the position INEO (me for instance) could officialy take on the project. Joshua and Paul proposed me to be Quality Assurance (QA) manager, and I personnaly accepted this role for a two months try. I've just asked my INEO managers for a formal validation. Have you head back? Can we announce it as official?
On IRC, the definition about QA manager was discussed, because everybody doesn't define it the same way. Paul said "responsible for heavy testing" and I answered I did not agree with this point of view, not in OSS model. thd asked what difference I made between open and closed source QA. Let's answer this question by mail, as requested by Russ. See details on IRC logs [1]. Right ... see comments below ...
I first answer to the question "what difference I make between open and closed source QA", then I'll describe what is my vision of QA manager for an OSS project like Koha.
QA, proprietay Vs OSS =====================
I've worked on the two sides of softwares : proprietary and OSS. I think I understand the advantages and drawbacks of each model, concerning QA.
In a proprietary model, the software *must* be /stable/ when reaching customers. In my vision, /stable/ means:
- without bug that may corrupt customers data - without blocking bug
Consequently, a proprietary software vendor needs to have a QA team. This team assures the stability of releases. No right to mistakes because financial penalties will be requested by customers.
OSS model is different because it *involves* customer on QA. This is a very important point in my opinion. An OSS project must provide efficient tools to communicate with core dev team. Customers/users are willing to propose patches.
OSS model partially moves QA to customers. Not completely of course, and I'll describe muy vision of QA management later. I'm sure Koha developers have all read Eric Raymond essay "The Cathedral & the Bazaar" [2] (if not, I encourage the reading). In this essay, you can read the summarize of Linus's law : "Given enough eyeballs, all bugs are shallow.". Koha team will never have enough eyeballs, this is why customers must be involved in bug tracking, from notification to test & validation.
Having worked nearly 3 years in a proprietary software company, I can assure you that QA is one of the most critical task, and very often underestimated since it costs a lot while not giving back money directly. QA team needs to grow at the same speed as development team output, and even more because complexity increases more than linearly.
OSS offers a solution to assure software quality without costing a lot more. Indeed, the more customers, the more requested features, the more tests required. If customers are involved in QA process, the number of testers will increase the same as the number of features. My conclusion is that in OSS, QA dedicated team don't need to be as big as development team.
My explanations are very theoretical, in the reality OSS vendors have to "release often, release early" [2] and involve their customer at an early stage of deployment, not only before production deployment. This is why HEAD builds, 3.0-RCx are necessary. I think some customers will accept to play the role of beta tester. I ask Paul, Joshua, Russ and other Koha services vendor to give their opinion. I think in general you're right, our customers often play the role of beta testers for the specific modules that we are developing for them. However, I can't tell you how many times a developer has written something that works perfectly for _their_ client, but it also breaks some function that someone else's client needs (for instance, budget- based acquisitions have been broken no less than 3 times).
So, what this means is that the customer can't be the only beta tester. We need someone to take a broad overview and make sure _all_ functions are working before a release. Paul, Chris, Russ, anyone have any further comments on this?
My QA manager vision on Koha ============================
I come to my vision of QA manager on Koha project. The most important of the QA manager is to know the content of the bug tracker:
1. being notified on every update 2. dispatch issue to developers (with developer authorization) 3. check that each issue is assigned/treated 4. reduce time between bug notification and its closure
It is important that no bug remain a too long time unassigned/untreated. If this happens, customer won't feel useful to involve herself. A bug rapidly corrected with a efficient communication between reporter and developer will encourage customers to report bugs, thus improving Koha quality in the long run. 100% agreed.
An additionnal task of the QA manager is Bug Squashing Party (BSP) coordination. I need to be explained how Koha worked on this in the past. Also a great idea. I was holding weekly BSP's a while back but have gotten out of the habit of scheduling them -- feel free to start this asap...it might also help us to get back to using bugzilla again.
Cheers, -- Joshua Ferraro VENDOR SERVICES FOR OPEN-SOURCE SOFTWARE President, Technology migration, training, maintenance, support LibLime Featuring Koha Open-Source ILS jmf@liblime.com |Full Demos at http://liblime.com/koha |1(888)KohaILS
Joshua Ferraro a écrit :
I think in general you're right, our customers often play the role of beta testers for the specific modules that we are developing for them. However, I can't tell you how many times a developer has written something that works perfectly for _their_ client, but it also breaks some function that someone else's client needs (for instance, budget- based acquisitions have been broken no less than 3 times).
So, what this means is that the customer can't be the only beta tester. We need someone to take a broad overview and make sure _all_ functions are working before a release.
Paul, Chris, Russ, anyone have any further comments on this?
yes : this will become harder and harder, as we will have more and more libraries using Koha.That'a also why I strongly suggest to my costomers to use an official Koha release, and not use local templates for librarian interface. And i will continue this way, as it proves that it's the best one to find where a bug comes from : it always comes from Koha, not from the local templates ! -- Paul POULAIN et Henri Damien LAURENT Consultants indépendants en logiciels libres et bibliothéconomie (http://www.koha-fr.org)
On Sat, Apr 01, 2006 at 06:33:00PM -0800, Joshua Ferraro wrote:
So, what this means is that the customer can't be the only beta tester. We need someone to take a broad overview and make sure _all_ functions are working before a release.
Paul, Chris, Russ, anyone have any further comments on this?
If Koha functionality is kept inside modules, then unit tests can be written to verify documented functionality is not broken. -kolibrie
On Tue, 21 Mar 2006 14:32:51 +0100 Pierrick LE GALL <pierrick@koha-fr.org> wrote:
I've just asked my INEO managers for a formal validation.
Of course, my managers gave me the validation, sorry for the long wait... First tasks I'll assume as QA manager: - first Bug Squashing Party on April 18th, 2006 - precisions on Coding Guidelines when the new wiki is online Bye -- Pierrick LE GALL INEO media system
participants (5)
-
Joshua Ferraro -
MJ Ray -
Nathan Gray -
Paul POULAIN -
Pierrick LE GALL