BibLibre strategy for Koha 3.4 and next versions
Hi, In this mail, I want to share with you our situation, our problems, and our ideas. You'll be probably surprised by this mail. Please consider it's a good news. It's not a threat - not my cup of tea (frenchism suspected) - . But I think things must be done this way, so I share it with you, and I'm impatiently waiting for koha-community reaction & thoughts. BibLibre situation =========== We are very successful. Koha is really a success in France. One french university is now live, we are on the way to migrate a very large public library (500k items), 3 other universities should go live in September, and a lot of business coming. We are 11, are on our way to hire 3 new ppl, have on board Philippe Chabanon, that is very experienced in managing a company, is fully enthusiastic about free softwares and community, and knows a lot of people in the french library world. Is everything perfect ? nope. Our problems ========= Organizational problem ------------ Some of our clients sponsor new developments. Those are done under "marché public" rules, that usually define a very strict date (if we fail, then we have to pay fines, based on how many days late we are !). That's why we started 3 different branches for our 3 different major sponsored works (Aix-Marseille, Nîmes, Lyon 3, you can see everything on git.biblibre.com). Everything that was in Aix-Marseille branch is now in koha 3.2 . But we have a lot of things done for Lyon 3 (that is live already) and Nimes (that will go live in early july) that are not in koha 3.2 Our main short-term problem is that koha 3.2 is very (very) late. It has been announced one year ago ! We did a lot of testing for Lyon 3 before putting them live. Some of them resulted in some bugfixes that could/should have been submitted to koha mainstream. But, we couldn't. Thus we face a situation where someone fixes in official 3.2 some things we already fixed in our Lyon3 branch (shame on us, but really, we couldn't find time. Delivery to Lyon3 was on time, but is was shorter than short) The situation is the same with Nimes : they will probably be live (with 3.2 + Lyon3 + Nimes sponsored stuff) before 3.2 is released -going live planned for july 1st, Galen, we challenge you ;-) -. So, frankly, dealing with our contracts and the official version of Koha has been a pain, and has resulted in a "virtual fork" for Lyon3 (and in 1 month for Nimes). We strongly want to submit our code to the main trunk, but dealing with a so-long-delayed version is really a nightmare (and I understand PTFS deploying "Harley" -even if i share community opinion about the version being advertised when released in a way it can be understood as an official version-). We must find a solution to this situation, or the "virtual fork" will result in a "real fork" that we strongly want to avoid ! As of today, we have something like 400 patches to integrate into koha main trunk ! Technical problem ----------- With some customers having more than 1 million items (Aix-Marseille : 1247686), and with more and more customers, we reach some limits with the structure of Koha, in term of performances and stability/maintainability. Chris suggested some improvements with koha 3.4, and we have some complementary ideas, I want to share with you, and would like to implement ASAP. Ideas/proposal ======= Technical : Koha architecture --------------------------------- At the begining of may 2010, we staff had a meeting at Marseille to know if it would be possible to make koha easier to maintain and evolve, we are now doing some experiment to split koha into several layers that provides completely independant set of tasks based on the MVC architecture (http://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller). Koha can be seen as a stack with communication between the closest layers. 4 layers at all (does it make you remember TCP/IP? ;) ) VIEW: interactions with different clients ( browser, REST, XMPP, Z3950, ... ) ?? CONTROLLER: Controls the model by transmitting the CRUD tasks requested by the VIEW ?? MODEL: Business objects, as close as possible to the real world ?? PERSISTENCE: Make the Business objects persistent Plus, we expect to use external tools as often as possible to take experience of other communities and reduce the amount of code we have to debug by ourselves. We think tools exists for every layers, actually we just have to choose between them: View: ------- HTML::Template, Template Toolkit 2, Template::Declare, SOAP::Lite, any REST, XMPP, Z3950, SRU/SRW wrapper, ... (Infobot ;)) Controller: ---------- Dancer, Mojo, The Controller parts of existing MVC frameworks ( Jifty, Catalyst, ...), ... Model: ------- Any OO Framework ( prefer those which comes with persistence extension as MooseX::Storage, KiokuDB, ... choose one ) Persistence: ----------- DBIx::Class, Document DB clients, SQL::Abstract, Jifty::DBI, .... Ideally, the communication between each layers must be specified, provable by unit testing, and well documented ( pods in the t/* ?). Today, Koha works in CGI mode, so there is no persistence at all. And having no persistence means we can't implement the MVC model -some tests we did with OO frameworks show very poor performances-, and won't be able to improve performance heavily -the main loss of perf in koha as of today is loading datas that are almost static & should be available immediatly : systempreferences & all the admin tables -. So: 1st step => remove CGI, to have persistent code, enabling persistent DB connection. We did some tests with Dancer, that are very encouraring, we will send a mail about that soon. 2nd step => Once we will be OK with dancer to get rid of CGI => head to persistence & OO framework. We vote for Moose & KiokuDB, but we still have POC to write to confirm the idea. 3rd step =>view part, by moving to T::T (POC still to do to confirm the idea) It may look like a koha rewrite but it's not: every parts would be added step by step into koha by wrapping the new design behind the old API during a transition phase. An example of current code sub GetBranchesInCategory($) { my ($categorycode) = @_; my @branches; my $dbh = C4::Context->dbh(); my $sth = $dbh->prepare( "SELECT b.branchcode FROM branchrelations r, branches b where r.branchcode=b.branchcode and r.categorycode=?" ); $sth->execute($categorycode); while ( my $branch = $sth->fetchrow ) { push @branches, $branch; } $sth->finish(); return ( \@branches ); } would became transition code: sub GetBranchesInCategory($) { my $id = shift or return; # - Category would be the Business object with a member transition (which can be # deprecated at the end of transition) that encapsulate every # persistance specific data used in the C4::* api # - the find method could be delegated to the DBIx::Class find that returns the # record associated to the id # returns if ( my $category = Category->transition->find($id) ) { # codes to deal with branches comes de facto with tools as DBIx::Class # it returns the list of branches for the category $category->branches; } else { undef } } You can find in attachement a small graph with all our ideas, and the beginning of a timeline. ppl reading carefully will see a "changing indexing system (solr)". 4 years after starting with it, we still find zebra being a nightmare, with a lot of problems for unicode (with ICU). We did some tests with solr, and they are very encouraging. We will provide the results soon too. But i'm really enthusiast about what we did in a few hours ! Organizational : Koha team ------------------------------- Koha team organisation ............... If we (BibLibre & the Koha community) want to be able to implement such a roadmap (that also includes some new functionnality, as customers sponsors some...), we must have a heavily organised team. I think we must : * manage release by date (not by features). This is probably the more important part for us (see our "marché public" problem). The timing of the versions would be defined by the companies contributing the most and synched with sponsored devs roadmap. That will be tricky to organize when there are more than one company involved, but, for instance, BibLibre is the largest contributor ( PTFS: you're welcomed to enter the thread ! ). In fact, "a koha 3.4" could be released in a few weeks (look at out master branch, it contains 3.2 + Lyon3 + Nimes sponsored work). A "koha 3.6" could be released in december/february,... a regular release, that's something that is probably "non-negociable" for us. * define very clear coding guidelines & be very stricts on their use : everything that does not respect the guidelines must be politely but firmly rejected. * have a very efficient patch workflow : a patch can't be in the wild for weeks. The submitter must have an answer in a very short timeline (is 48H possible ?). ie : We can't rely on only one person to push patches. We must have back-ups for major roles like RM & "patch pusher". * we could divide Koha in functionnal (serials / acquisition / cataloguing / ...) or technical (perfs / SQL structure / view layer / indexing engine / ...) parts, with some working group for each of them, and something like a monthly report of the WG (with a weekly internal meeting) * write tests ! technical tests (.t) and functional tests (selenium ?) Our involvement (BibLibre) : .................... We want to improve our involvement in Koha, by dedicating Henri-Damien to work at 75% for Koha and be the link between the work done internally by BibLibre, and the work done by everyone else. This mean : * publish our RFPs as soon as possible * gather all the work done every month and communicate about that on the wiki and blog and translate on koha-fr.org * manage patches coming out from BibLibre, and coming out from everyone else (to deal as soon as possible with any problem) I know we have elected a team recently, and I know chris will do an awesome job as RM (I don't know for others, I didn't saw them in their responsability yet. For chris, i'm the team for 8 years, I know who he is ;-) ), but I think hdl should have a responsability more important than "bug wrangler" to reflect the time he (will) dedicate to the project. Is something like "co-RM" possible ? any other idea ? Any other idea ? Next step ----------- We are *really* willing to implement this roadmap and way of doing things. The goal of this mail is to ask everybody involved in Koha development about their opinion: we want our proposal to be a chance for Koha (new deal ?). On the technical side, stay tuned. Some POC will fall in the next weeks !!! On the organizational side, the ball is on your side: let's go to update/improve our organization ! -- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc Tel : (33) 4 91 81 35 08
Hi Paul There is an IRC meeting today, in 3 hours time .. but this topic is much to big for that, so I propose we leave it off the agenda for that, and let people think about the ideas raised for a bit, and then do replies. Speaking for myself nothing in the email strikes me as bad news, I do think time based releases cause their own set of problems, and as RM elect I think if we decided on that we would certainly have to become a LOT stricter about patches. IE, we would need to reject more and faster .. which might not be a bad thing anyway. I also don't think bug wrangler is a small job .. .anyone who thinks that hasnt looked in bugzilla recently ;-) Some of the main tasks of a wrangler are just what you describe Henri-Damien as doing, ie, managing integration of bug fixes and new features. One thing that would allow patches to be accepted a lot faster would be making every patch need to contain a working test. One thing I am considering is borrowing from the Java people and getting Hudson up and running http://wiki.hudson-ci.org/display/HUDSON/Meet+Hudson Of course, we need unit tests to make this useful. As for the technical part of managing many branches in git, we have significant experience of that at work, in the team I work on every single feature, or bug, is in its' own branch and they are merged into testing releases (we have literally hundreds of branches). Git makes this easy, so I think the use of many more feature branches at git.koha-community.org will make merging a lot easier, this means even virtual forks are only just a branch. As to the speed of pushing patches, test cases and good commit messages would improve that immensely. But I am confident that with the help of a QA manager, which poor Galen never had, we can accept or reject patches in a timely manner. The better your commit message the easier that gets, if we have to read all the code just to try and figure out what its trying to do, then to figure out if it is actually doing it .. that will slow things down immensely. So as RM I accept the challenge of faster releases, and throw back a challenge of better patches :) Chris 2010/6/2 Paul Poulain <paul.poulain@biblibre.com>:
Hi,
In this mail, I want to share with you our situation, our problems, and our ideas. You'll be probably surprised by this mail. Please consider it's a good news. It's not a threat - not my cup of tea (frenchism suspected) - . But I think things must be done this way, so I share it with you, and I'm impatiently waiting for koha-community reaction & thoughts.
BibLibre situation =========== We are very successful. Koha is really a success in France. One french university is now live, we are on the way to migrate a very large public library (500k items), 3 other universities should go live in September, and a lot of business coming. We are 11, are on our way to hire 3 new ppl, have on board Philippe Chabanon, that is very experienced in managing a company, is fully enthusiastic about free softwares and community, and knows a lot of people in the french library world. Is everything perfect ? nope.
Our problems ========= Organizational problem ------------ Some of our clients sponsor new developments. Those are done under "marché public" rules, that usually define a very strict date (if we fail, then we have to pay fines, based on how many days late we are !). That's why we started 3 different branches for our 3 different major sponsored works (Aix-Marseille, Nîmes, Lyon 3, you can see everything on git.biblibre.com). Everything that was in Aix-Marseille branch is now in koha 3.2 . But we have a lot of things done for Lyon 3 (that is live already) and Nimes (that will go live in early july) that are not in koha 3.2
Our main short-term problem is that koha 3.2 is very (very) late. It has been announced one year ago ! We did a lot of testing for Lyon 3 before putting them live. Some of them resulted in some bugfixes that could/should have been submitted to koha mainstream. But, we couldn't. Thus we face a situation where someone fixes in official 3.2 some things we already fixed in our Lyon3 branch (shame on us, but really, we couldn't find time. Delivery to Lyon3 was on time, but is was shorter than short) The situation is the same with Nimes : they will probably be live (with 3.2 + Lyon3 + Nimes sponsored stuff) before 3.2 is released -going live planned for july 1st, Galen, we challenge you ;-) -.
So, frankly, dealing with our contracts and the official version of Koha has been a pain, and has resulted in a "virtual fork" for Lyon3 (and in 1 month for Nimes). We strongly want to submit our code to the main trunk, but dealing with a so-long-delayed version is really a nightmare (and I understand PTFS deploying "Harley" -even if i share community opinion about the version being advertised when released in a way it can be understood as an official version-). We must find a solution to this situation, or the "virtual fork" will result in a "real fork" that we strongly want to avoid !
As of today, we have something like 400 patches to integrate into koha main trunk !
Technical problem ----------- With some customers having more than 1 million items (Aix-Marseille : 1247686), and with more and more customers, we reach some limits with the structure of Koha, in term of performances and stability/maintainability. Chris suggested some improvements with koha 3.4, and we have some complementary ideas, I want to share with you, and would like to implement ASAP.
Ideas/proposal ======= Technical : Koha architecture --------------------------------- At the begining of may 2010, we staff had a meeting at Marseille to know if it would be possible to make koha easier to maintain and evolve, we are now doing some experiment to split koha into several layers that provides completely independant set of tasks based on the MVC architecture (http://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller).
Koha can be seen as a stack with communication between the closest layers. 4 layers at all (does it make you remember TCP/IP? ;) )
VIEW: interactions with different clients ( browser, REST, XMPP, Z3950, ... ) ↓↑ CONTROLLER: Controls the model by transmitting the CRUD tasks requested by the VIEW ↓↑ MODEL: Business objects, as close as possible to the real world ↓↑ PERSISTENCE: Make the Business objects persistent
Plus, we expect to use external tools as often as possible to take experience of other communities and reduce the amount of code we have to debug by ourselves. We think tools exists for every layers, actually we just have to choose between them:
View: ------- HTML::Template, Template Toolkit 2, Template::Declare, SOAP::Lite, any REST, XMPP, Z3950, SRU/SRW wrapper, ... (Infobot ;))
Controller: ---------- Dancer, Mojo, The Controller parts of existing MVC frameworks ( Jifty, Catalyst, ...), ...
Model: ------- Any OO Framework ( prefer those which comes with persistence extension as MooseX::Storage, KiokuDB, ... choose one )
Persistence: ----------- DBIx::Class, Document DB clients, SQL::Abstract, Jifty::DBI, ....
Ideally, the communication between each layers must be specified, provable by unit testing, and well documented ( pods in the t/* ?).
Today, Koha works in CGI mode, so there is no persistence at all. And having no persistence means we can't implement the MVC model -some tests we did with OO frameworks show very poor performances-, and won't be able to improve performance heavily -the main loss of perf in koha as of today is loading datas that are almost static & should be available immediatly : systempreferences & all the admin tables -.
So: 1st step => remove CGI, to have persistent code, enabling persistent DB connection. We did some tests with Dancer, that are very encouraring, we will send a mail about that soon. 2nd step => Once we will be OK with dancer to get rid of CGI => head to persistence & OO framework. We vote for Moose & KiokuDB, but we still have POC to write to confirm the idea. 3rd step =>view part, by moving to T::T (POC still to do to confirm the idea)
It may look like a koha rewrite but it's not: every parts would be added step by step into koha by wrapping the new design behind the old API during a transition phase. An example of current code
sub GetBranchesInCategory($) { my ($categorycode) = @_; my @branches; my $dbh = C4::Context->dbh(); my $sth = $dbh->prepare( "SELECT b.branchcode FROM branchrelations r, branches b where r.branchcode=b.branchcode and r.categorycode=?" ); $sth->execute($categorycode); while ( my $branch = $sth->fetchrow ) { push @branches, $branch; } $sth->finish(); return ( \@branches ); }
would became transition code:
sub GetBranchesInCategory($) { my $id = shift or return;
# - Category would be the Business object with a member transition (which can be # deprecated at the end of transition) that encapsulate every # persistance specific data used in the C4::* api # - the find method could be delegated to the DBIx::Class find that returns the # record associated to the id
# returns if ( my $category = Category->transition->find($id) ) { # codes to deal with branches comes de facto with tools as DBIx::Class # it returns the list of branches for the category $category->branches; } else { undef } }
You can find in attachement a small graph with all our ideas, and the beginning of a timeline.
ppl reading carefully will see a "changing indexing system (solr)". 4 years after starting with it, we still find zebra being a nightmare, with a lot of problems for unicode (with ICU). We did some tests with solr, and they are very encouraging. We will provide the results soon too. But i'm really enthusiast about what we did in a few hours !
Organizational : Koha team ------------------------------- Koha team organisation ............... If we (BibLibre & the Koha community) want to be able to implement such a roadmap (that also includes some new functionnality, as customers sponsors some...), we must have a heavily organised team. I think we must : * manage release by date (not by features). This is probably the more important part for us (see our "marché public" problem). The timing of the versions would be defined by the companies contributing the most and synched with sponsored devs roadmap. That will be tricky to organize when there are more than one company involved, but, for instance, BibLibre is the largest contributor ( PTFS: you're welcomed to enter the thread ! ). In fact, "a koha 3.4" could be released in a few weeks (look at out master branch, it contains 3.2 + Lyon3 + Nimes sponsored work). A "koha 3.6" could be released in december/february,... a regular release, that's something that is probably "non-negociable" for us. * define very clear coding guidelines & be very stricts on their use : everything that does not respect the guidelines must be politely but firmly rejected. * have a very efficient patch workflow : a patch can't be in the wild for weeks. The submitter must have an answer in a very short timeline (is 48H possible ?). ie : We can't rely on only one person to push patches. We must have back-ups for major roles like RM & "patch pusher". * we could divide Koha in functionnal (serials / acquisition / cataloguing / ...) or technical (perfs / SQL structure / view layer / indexing engine / ...) parts, with some working group for each of them, and something like a monthly report of the WG (with a weekly internal meeting) * write tests ! technical tests (.t) and functional tests (selenium ?)
Our involvement (BibLibre) : .................... We want to improve our involvement in Koha, by dedicating Henri-Damien to work at 75% for Koha and be the link between the work done internally by BibLibre, and the work done by everyone else. This mean :
publish our RFPs as soon as possible gather all the work done every month and communicate about that on the wiki and blog and translate on koha-fr.org manage patches coming out from BibLibre, and coming out from everyone else (to deal as soon as possible with any problem)
I know we have elected a team recently, and I know chris will do an awesome job as RM (I don't know for others, I didn't saw them in their responsability yet. For chris, i'm the team for 8 years, I know who he is ;-) ), but I think hdl should have a responsability more important than "bug wrangler" to reflect the time he (will) dedicate to the project. Is something like "co-RM" possible ? any other idea ? Any other idea ?
Next step ----------- We are *really* willing to implement this roadmap and way of doing things. The goal of this mail is to ask everybody involved in Koha development about their opinion: we want our proposal to be a chance for Koha (new deal ?).
On the technical side, stay tuned. Some POC will fall in the next weeks !!! On the organizational side, the ball is on your side: let's go to update/improve our organization !
-- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc Tel : (33) 4 91 81 35 08
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
Le 02/06/2010 10:25, Chris Cormack a écrit :
Hi Paul
There is an IRC meeting today, in 3 hours time .. but this topic is much to big for that, so I propose we leave it off the agenda for that, and let people think about the ideas raised for a bit, and then do replies.
Hi chris, this mail was sent today so that people can react, the intent was not to debate of that during today's meeting (agenda is already online) but to raise our questions and tell our point of view. Anyway, decisions and actions will have to be taken forward.
Speaking for myself nothing in the email strikes me as bad news, I do think time based releases cause their own set of problems, and as RM elect I think if we decided on that we would certainly have to become a LOT stricter about patches. IE, we would need to reject more and faster .. which might not be a bad thing anyway. I also don't think bug wrangler is a small job .. .anyone who thinks that hasnt looked in bugzilla recently ;-) Some of the main tasks of a wrangler are just what you describe Henri-Damien as doing, ie, managing integration of bug fixes and new features.
One thing that would allow patches to be accepted a lot faster would be making every patch need to contain a working test. One thing I am considering is borrowing from the Java people and getting Hudson up and running http://wiki.hudson-ci.org/display/HUDSON/Meet+Hudson Of course, we need unit tests to make this useful.
As for the technical part of managing many branches in git, we have significant experience of that at work, in the team I work on every single feature, or bug, is in its' own branch and they are merged into testing releases (we have literally hundreds of branches). Git makes this easy, so I think the use of many more feature branches at git.koha-community.org will make merging a lot easier, this means even virtual forks are only just a branch.
The way koha code is now is making that troublesome since when we fix some circulation stuff or holding stuff, we have to touch many files, so that when rebasing, we have many conflicts. Moreover, since our customers ask for features and not only bug fixes, and want a kind of relesase when they are delivered, we have to merge both devs and bugfixes in order to achieve global stability and coherence betweeen devs and bug fixes. Some devs we did also helped us in fixing bugs quicker, simpler and better. So that if we ported that to master back, on order to ensure that nothing is broken, i need more than just a few hours, would require me not just cherry-picking but also some kind of refactoring and then would introduce some risks both in koha-community master branch (overlooked bug) and in our branch when we rebase on koha-community master branch. This is why we also propose some new tools in order to improve coding guidelines and eventually have some way to automate testing.
As to the speed of pushing patches, test cases and good commit messages would improve that immensely. But I am confident that with the help of a QA manager, which poor Galen never had, we can accept or reject patches in a timely manner. The better your commit message the easier that gets, if we have to read all the code just to try and figure out what its trying to do, then to figure out if it is actually doing it .. that will slow things down immensely.
So as RM I accept the challenge of faster releases, and throw back a challenge of better patches :)
Well, there are some good exemples, and we are getting better, but it is also quite difficult to raise the cursor to the good point and when we do a quick followup, difficult to resist the temptation of using Followup.... And when we are having six months delay on integration of our branch, it is difficult to re process the whole history. It is the work of release manager, QA manager, Bug wrangler, and eventually, all the subscribers to koha-patches to ask for details. But what about branches to come ? quickly sent... my 2 cents. -- Henri-Damien LAURENT
Chris
2010/6/2 Paul Poulain <paul.poulain@biblibre.com>:
Hi,
In this mail, I want to share with you our situation, our problems, and our ideas. You'll be probably surprised by this mail. Please consider it's a good news. It's not a threat - not my cup of tea (frenchism suspected) - . But I think things must be done this way, so I share it with you, and I'm impatiently waiting for koha-community reaction & thoughts.
BibLibre situation =========== We are very successful. Koha is really a success in France. One french university is now live, we are on the way to migrate a very large public library (500k items), 3 other universities should go live in September, and a lot of business coming. We are 11, are on our way to hire 3 new ppl, have on board Philippe Chabanon, that is very experienced in managing a company, is fully enthusiastic about free softwares and community, and knows a lot of people in the french library world. Is everything perfect ? nope.
Our problems ========= Organizational problem ------------ Some of our clients sponsor new developments. Those are done under "marché public" rules, that usually define a very strict date (if we fail, then we have to pay fines, based on how many days late we are !). That's why we started 3 different branches for our 3 different major sponsored works (Aix-Marseille, Nîmes, Lyon 3, you can see everything on git.biblibre.com). Everything that was in Aix-Marseille branch is now in koha 3.2 . But we have a lot of things done for Lyon 3 (that is live already) and Nimes (that will go live in early july) that are not in koha 3.2
Our main short-term problem is that koha 3.2 is very (very) late. It has been announced one year ago ! We did a lot of testing for Lyon 3 before putting them live. Some of them resulted in some bugfixes that could/should have been submitted to koha mainstream. But, we couldn't. Thus we face a situation where someone fixes in official 3.2 some things we already fixed in our Lyon3 branch (shame on us, but really, we couldn't find time. Delivery to Lyon3 was on time, but is was shorter than short) The situation is the same with Nimes : they will probably be live (with 3.2 + Lyon3 + Nimes sponsored stuff) before 3.2 is released -going live planned for july 1st, Galen, we challenge you ;-) -.
So, frankly, dealing with our contracts and the official version of Koha has been a pain, and has resulted in a "virtual fork" for Lyon3 (and in 1 month for Nimes). We strongly want to submit our code to the main trunk, but dealing with a so-long-delayed version is really a nightmare (and I understand PTFS deploying "Harley" -even if i share community opinion about the version being advertised when released in a way it can be understood as an official version-). We must find a solution to this situation, or the "virtual fork" will result in a "real fork" that we strongly want to avoid !
As of today, we have something like 400 patches to integrate into koha main trunk !
Technical problem ----------- With some customers having more than 1 million items (Aix-Marseille : 1247686), and with more and more customers, we reach some limits with the structure of Koha, in term of performances and stability/maintainability. Chris suggested some improvements with koha 3.4, and we have some complementary ideas, I want to share with you, and would like to implement ASAP.
Ideas/proposal ======= Technical : Koha architecture --------------------------------- At the begining of may 2010, we staff had a meeting at Marseille to know if it would be possible to make koha easier to maintain and evolve, we are now doing some experiment to split koha into several layers that provides completely independant set of tasks based on the MVC architecture (http://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller).
Koha can be seen as a stack with communication between the closest layers. 4 layers at all (does it make you remember TCP/IP? ;) )
VIEW: interactions with different clients ( browser, REST, XMPP, Z3950, ... ) ↓↑ CONTROLLER: Controls the model by transmitting the CRUD tasks requested by the VIEW ↓↑ MODEL: Business objects, as close as possible to the real world ↓↑ PERSISTENCE: Make the Business objects persistent
Plus, we expect to use external tools as often as possible to take experience of other communities and reduce the amount of code we have to debug by ourselves. We think tools exists for every layers, actually we just have to choose between them:
View: ------- HTML::Template, Template Toolkit 2, Template::Declare, SOAP::Lite, any REST, XMPP, Z3950, SRU/SRW wrapper, ... (Infobot ;))
Controller: ---------- Dancer, Mojo, The Controller parts of existing MVC frameworks ( Jifty, Catalyst, ...), ...
Model: ------- Any OO Framework ( prefer those which comes with persistence extension as MooseX::Storage, KiokuDB, ... choose one )
Persistence: ----------- DBIx::Class, Document DB clients, SQL::Abstract, Jifty::DBI, ....
Ideally, the communication between each layers must be specified, provable by unit testing, and well documented ( pods in the t/* ?).
Today, Koha works in CGI mode, so there is no persistence at all. And having no persistence means we can't implement the MVC model -some tests we did with OO frameworks show very poor performances-, and won't be able to improve performance heavily -the main loss of perf in koha as of today is loading datas that are almost static & should be available immediatly : systempreferences & all the admin tables -.
So: 1st step => remove CGI, to have persistent code, enabling persistent DB connection. We did some tests with Dancer, that are very encouraring, we will send a mail about that soon. 2nd step => Once we will be OK with dancer to get rid of CGI => head to persistence & OO framework. We vote for Moose & KiokuDB, but we still have POC to write to confirm the idea. 3rd step =>view part, by moving to T::T (POC still to do to confirm the idea)
It may look like a koha rewrite but it's not: every parts would be added step by step into koha by wrapping the new design behind the old API during a transition phase. An example of current code
sub GetBranchesInCategory($) { my ($categorycode) = @_; my @branches; my $dbh = C4::Context->dbh(); my $sth = $dbh->prepare( "SELECT b.branchcode FROM branchrelations r, branches b where r.branchcode=b.branchcode and r.categorycode=?" ); $sth->execute($categorycode); while ( my $branch = $sth->fetchrow ) { push @branches, $branch; } $sth->finish(); return ( \@branches ); }
would became transition code:
sub GetBranchesInCategory($) { my $id = shift or return;
# - Category would be the Business object with a member transition (which can be # deprecated at the end of transition) that encapsulate every # persistance specific data used in the C4::* api # - the find method could be delegated to the DBIx::Class find that returns the # record associated to the id
# returns if ( my $category = Category->transition->find($id) ) { # codes to deal with branches comes de facto with tools as DBIx::Class # it returns the list of branches for the category $category->branches; } else { undef } }
You can find in attachement a small graph with all our ideas, and the beginning of a timeline.
ppl reading carefully will see a "changing indexing system (solr)". 4 years after starting with it, we still find zebra being a nightmare, with a lot of problems for unicode (with ICU). We did some tests with solr, and they are very encouraging. We will provide the results soon too. But i'm really enthusiast about what we did in a few hours !
Organizational : Koha team ------------------------------- Koha team organisation ............... If we (BibLibre & the Koha community) want to be able to implement such a roadmap (that also includes some new functionnality, as customers sponsors some...), we must have a heavily organised team. I think we must : * manage release by date (not by features). This is probably the more important part for us (see our "marché public" problem). The timing of the versions would be defined by the companies contributing the most and synched with sponsored devs roadmap. That will be tricky to organize when there are more than one company involved, but, for instance, BibLibre is the largest contributor ( PTFS: you're welcomed to enter the thread ! ). In fact, "a koha 3.4" could be released in a few weeks (look at out master branch, it contains 3.2 + Lyon3 + Nimes sponsored work). A "koha 3.6" could be released in december/february,... a regular release, that's something that is probably "non-negociable" for us. * define very clear coding guidelines & be very stricts on their use : everything that does not respect the guidelines must be politely but firmly rejected. * have a very efficient patch workflow : a patch can't be in the wild for weeks. The submitter must have an answer in a very short timeline (is 48H possible ?). ie : We can't rely on only one person to push patches. We must have back-ups for major roles like RM & "patch pusher". * we could divide Koha in functionnal (serials / acquisition / cataloguing / ...) or technical (perfs / SQL structure / view layer / indexing engine / ...) parts, with some working group for each of them, and something like a monthly report of the WG (with a weekly internal meeting) * write tests ! technical tests (.t) and functional tests (selenium ?)
Our involvement (BibLibre) : .................... We want to improve our involvement in Koha, by dedicating Henri-Damien to work at 75% for Koha and be the link between the work done internally by BibLibre, and the work done by everyone else. This mean :
publish our RFPs as soon as possible gather all the work done every month and communicate about that on the wiki and blog and translate on koha-fr.org manage patches coming out from BibLibre, and coming out from everyone else (to deal as soon as possible with any problem)
I know we have elected a team recently, and I know chris will do an awesome job as RM (I don't know for others, I didn't saw them in their responsability yet. For chris, i'm the team for 8 years, I know who he is ;-) ), but I think hdl should have a responsability more important than "bug wrangler" to reflect the time he (will) dedicate to the project. Is something like "co-RM" possible ? any other idea ? Any other idea ?
Next step ----------- We are *really* willing to implement this roadmap and way of doing things. The goal of this mail is to ask everybody involved in Koha development about their opinion: we want our proposal to be a chance for Koha (new deal ?).
On the technical side, stay tuned. Some POC will fall in the next weeks !!! On the organizational side, the ball is on your side: let's go to update/improve our organization !
-- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc Tel : (33) 4 91 81 35 08
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
Any reason not to move to a 4 week release cycle? With alternate ones getting "better" testing. It would mean automating a lot of things -- but doable and worth it in any case. -reed
Le 02/06/2010 12:22, Reed Wade a écrit :
Any reason not to move to a 4 week release cycle? With alternate ones getting "better" testing.
It would mean automating a lot of things -- but doable and worth it in any case.
^^The main reason is this one. and no doubt that it IS BOTH doable AND worth.... But in what time and who would do that... BibLibre is willing to devote time for that... But this surely is a community effort with development and should be managed as such. maybe there should be teams along with the assignees of bugs which could make smaller patches, and meetings so as to report progess. -- Henri-Damien LAURENT
On Thu, Jun 3, 2010 at 3:07 AM, LAURENT Henri-Damien <laurenthdl@alinto.com> wrote:
Le 02/06/2010 12:22, Reed Wade a écrit :
Any reason not to move to a 4 week release cycle? With alternate ones getting "better" testing.
It would mean automating a lot of things -- but doable and worth it in any case.
^^The main reason is this one. and no doubt that it IS BOTH doable AND worth.... But in what time and who would do that... BibLibre is willing to devote time for that... But this surely is a community effort with development and should be managed as such. maybe there should be teams along with the assignees of bugs which could make smaller patches, and meetings so as to report progess.
(4 weeks is probably too short but...) Yes, getting the science and organisation correct would be key. Something ChrisC has alluded to in emails and IRC is the scheme we have developed at Catalyst for a project he and I and (5-8 others) work on. It is a long running development effort for a high scale news web site. It has a similar architecture and larger code base than Koha. We use git and there's a lot of perl. It seems likely there are some bits of code and techniques that could be applied to Koha release mgmt. We have a 4 (really 1-6) week release cycle which is feature and date driven (customer might need new thing in time for national elections or sporting event and they don't tend to plan too far ahead). We use debian packages for all code deployments and DB updates. We use a system similar to bugzilla for tracking code development. Each feature or bug fix gets a branch named according to the ticket number. We have an automated process which merges all branches for the next release (it queries the issue tracker to know what tickets are in what release) and emails the authors if there's a conflict. This runs several times a day. Instructions embedded in the ticket comments are used to sort dependencies and additional / alternate branches if needed for conflict resolution. We have a similar automated process which performs the merge then creates debian packages for our testing environment. It emails the results to the release manage who reviews and sets the status of updated features so QA staff know what they can now test (this should be automated but isn't just yet because I like to review these still). We have a pre-production autobuild set up which is a duplicate of all that but it only takes branches that QA have signed off on (examines ticket status). This scheme has been extraordinarily satisfying and effective and allows us to spend more time on real work instead of the mechanics of the process. Production deployments used to be long bad days and now they tend to be uneventful. Everyone is happier. Release management is now time spent considering product features instead of the mechanics of integration and testing. The key benefits we've gained are: - at any moment you can play with what would be the release if you decided to release at that moment - developers become responsible for fixing code conflicts or other blocks to their branch getting into a release and without hardly any time needed by the release manager (so very low latency for fixes) to wrangle that because it's automated emails telling the developer something went wrong -- How to apply to Koha? There are differences. We are a small group, mostly in the same office. Our customer is a 5 minute walk down the street from us. I was about to say that our trade-offs and priorities are clear -- but that wasn't true before we had the system described above. It's allowed us and the customer to focus on keeping things clear because they've seen the value of things like having a feature freeze date that doesn't change unless the release date changes. Some key things that could be taken-- - the auto-build and merge and report and package on a test machine for easy QA system -- this is something we should be able to adapt to bugzilla and make available for the project - the notion of naming branches according to bug/feature id -- we've found this to be very powerful Chris or Lars may have something to add. -reed
On 3 June 2010 11:38, Reed Wade <reed@catalyst.net.nz> wrote:
On Thu, Jun 3, 2010 at 3:07 AM, LAURENT Henri-Damien <laurenthdl@alinto.com> wrote:
Le 02/06/2010 12:22, Reed Wade a écrit :
Any reason not to move to a 4 week release cycle? With alternate ones getting "better" testing.
It would mean automating a lot of things -- but doable and worth it in any case.
^^The main reason is this one. and no doubt that it IS BOTH doable AND worth.... But in what time and who would do that... BibLibre is willing to devote time for that... But this surely is a community effort with development and should be managed as such. maybe there should be teams along with the assignees of bugs which could make smaller patches, and meetings so as to report progess.
(4 weeks is probably too short but...)
Yes, getting the science and organisation correct would be key.
Something ChrisC has alluded to in emails and IRC is the scheme we have developed at Catalyst for a project he and I and (5-8 others) work on. It is a long running development effort for a high scale news web site. It has a similar architecture and larger code base than Koha. We use git and there's a lot of perl. It seems likely there are some bits of code and techniques that could be applied to Koha release mgmt.
We have a 4 (really 1-6) week release cycle which is feature and date driven (customer might need new thing in time for national elections or sporting event and they don't tend to plan too far ahead).
We use debian packages for all code deployments and DB updates.
We use a system similar to bugzilla for tracking code development. Each feature or bug fix gets a branch named according to the ticket number.
We have an automated process which merges all branches for the next release (it queries the issue tracker to know what tickets are in what release) and emails the authors if there's a conflict. This runs several times a day. Instructions embedded in the ticket comments are used to sort dependencies and additional / alternate branches if needed for conflict resolution.
We have a similar automated process which performs the merge then creates debian packages for our testing environment. It emails the results to the release manage who reviews and sets the status of updated features so QA staff know what they can now test (this should be automated but isn't just yet because I like to review these still).
We have a pre-production autobuild set up which is a duplicate of all that but it only takes branches that QA have signed off on (examines ticket status).
This scheme has been extraordinarily satisfying and effective and allows us to spend more time on real work instead of the mechanics of the process. Production deployments used to be long bad days and now they tend to be uneventful. Everyone is happier. Release management is now time spent considering product features instead of the mechanics of integration and testing.
The key benefits we've gained are: - at any moment you can play with what would be the release if you decided to release at that moment - developers become responsible for fixing code conflicts or other blocks to their branch getting into a release and without hardly any time needed by the release manager (so very low latency for fixes) to wrangle that because it's automated emails telling the developer something went wrong
--
How to apply to Koha?
There are differences. We are a small group, mostly in the same office. Our customer is a 5 minute walk down the street from us.
I was about to say that our trade-offs and priorities are clear -- but that wasn't true before we had the system described above. It's allowed us and the customer to focus on keeping things clear because they've seen the value of things like having a feature freeze date that doesn't change unless the release date changes.
Some key things that could be taken--
- the auto-build and merge and report and package on a test machine for easy QA system -- this is something we should be able to adapt to bugzilla and make available for the project
- the notion of naming branches according to bug/feature id -- we've found this to be very powerful
Chris or Lars may have something to add.
I think the most important part of our system is indeed the separate branches per feature/bug This allows features to be removed and the release still happen, if they fail testing. Time based releases simply can not happen if we cant easily remove failing code. (Or not merge it as the case may be). Chris
On to, 2010-06-03 at 11:38 +1200, Reed Wade wrote:
Chris or Lars may have something to add.
This whole issue of how to structure development processes for Koha is pretty big, and I think it might be good to avoid changing too much too fast. For Koha 3.4, off the top of my head, I would suggest: * automatically measure test coverage for the test suite * reject patches that decrease test coverage * reject patches early for obviously low quality - bad commit messages - too many changes in one patch - automatic test suite does not pass - other things that make it hard for RM to review * keep the master branch always ready to release, as far as possible * possibly discuss changes on koha-devel before they are accepted - might apply only to non-trivial things, in the RM's opinion - discussion might help to find the best way of implementing feature * set up a test machine running the current tip of master - Debian packages would make this easy to update - available to everyone (via the web interface) - easy way to see if things work or not, and if the current version exhibits a bug or not That might all already be too much change, so implementing changes more slowly might be a good idea.
Had a thought randomly in the night about this thread, so I thought I'd resurrect it briefly. There had been some discussion about whether to base releases on features or fixed time schedules, or a combination thereof. I think a combination is probably the best, and would suggest the following: X.Y releases (like 3.2 or 3.4) being released on a features basis. That is, when a sufficient number of the RFCs for the release are met, the Release Manager says its time to publish the new version. Hopefully this will happen on a fairly regular basis, but we all know how development can go... X.Y.Z releases (like 3.2.1 or 3.2.2) being released on a fixed schedule. These updates include all the new features and bug fixes that the Quality Assurance Manager and Release Maintainer agree are stable and solid. There will be as many of these updates as there is time for before the next major release. I'm completely open to the periodicity of these releases. As Chris points out, this is only possible if bug fixes and features can be isolated from one another for integration into the timed releases. Developers publishing early and often will help with this, as well as maintaining Git repos with separate branches for various developments. The new git.koha-community.org site has the potential to allow various development groups to push code to their own branches in a safe and secure way, so that may be one way for teams working on longer-term projects to more easily integrate separate features. I believe this could also improve collaboration across development teams. Does this sound reasonable to folks? Am I missing any major points of consideration? Cheers, -Ian On Wed, Jun 2, 2010 at 7:38 PM, Reed Wade <reed@catalyst.net.nz> wrote:
On Thu, Jun 3, 2010 at 3:07 AM, LAURENT Henri-Damien <laurenthdl@alinto.com> wrote:
Le 02/06/2010 12:22, Reed Wade a écrit :
Any reason not to move to a 4 week release cycle? With alternate ones getting "better" testing.
It would mean automating a lot of things -- but doable and worth it in any case.
^^The main reason is this one. and no doubt that it IS BOTH doable AND worth.... But in what time and who would do that... BibLibre is willing to devote time for that... But this surely is a community effort with development and should be managed as such. maybe there should be teams along with the assignees of bugs which could make smaller patches, and meetings so as to report progess.
(4 weeks is probably too short but...)
Yes, getting the science and organisation correct would be key.
Something ChrisC has alluded to in emails and IRC is the scheme we have developed at Catalyst for a project he and I and (5-8 others) work on. It is a long running development effort for a high scale news web site. It has a similar architecture and larger code base than Koha. We use git and there's a lot of perl. It seems likely there are some bits of code and techniques that could be applied to Koha release mgmt.
We have a 4 (really 1-6) week release cycle which is feature and date driven (customer might need new thing in time for national elections or sporting event and they don't tend to plan too far ahead).
We use debian packages for all code deployments and DB updates.
We use a system similar to bugzilla for tracking code development. Each feature or bug fix gets a branch named according to the ticket number.
We have an automated process which merges all branches for the next release (it queries the issue tracker to know what tickets are in what release) and emails the authors if there's a conflict. This runs several times a day. Instructions embedded in the ticket comments are used to sort dependencies and additional / alternate branches if needed for conflict resolution.
We have a similar automated process which performs the merge then creates debian packages for our testing environment. It emails the results to the release manage who reviews and sets the status of updated features so QA staff know what they can now test (this should be automated but isn't just yet because I like to review these still).
We have a pre-production autobuild set up which is a duplicate of all that but it only takes branches that QA have signed off on (examines ticket status).
This scheme has been extraordinarily satisfying and effective and allows us to spend more time on real work instead of the mechanics of the process. Production deployments used to be long bad days and now they tend to be uneventful. Everyone is happier. Release management is now time spent considering product features instead of the mechanics of integration and testing.
The key benefits we've gained are: - at any moment you can play with what would be the release if you decided to release at that moment - developers become responsible for fixing code conflicts or other blocks to their branch getting into a release and without hardly any time needed by the release manager (so very low latency for fixes) to wrangle that because it's automated emails telling the developer something went wrong
--
How to apply to Koha?
There are differences. We are a small group, mostly in the same office. Our customer is a 5 minute walk down the street from us.
I was about to say that our trade-offs and priorities are clear -- but that wasn't true before we had the system described above. It's allowed us and the customer to focus on keeping things clear because they've seen the value of things like having a feature freeze date that doesn't change unless the release date changes.
Some key things that could be taken--
- the auto-build and merge and report and package on a test machine for easy QA system -- this is something we should be able to adapt to bugzilla and make available for the project
- the notion of naming branches according to bug/feature id -- we've found this to be very powerful
Chris or Lars may have something to add.
-reed _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
-- Ian Walls Lead Development Specialist ByWater Solutions ALA Booth # 817 Phone # (888) 900-8944 http://bywatersolutions.com ian.walls@bywatersolutions.com Twitter: @sekjal
I was actually thinking the other day (when WP 3.0 was released) and this model is probably something like what they do. Lots of little releases between each major release keeps me excited for the next time WordPress prompts me to upgrade :) Nicole 2010/6/18 Ian Walls <ian.walls@bywatersolutions.com>
Had a thought randomly in the night about this thread, so I thought I'd resurrect it briefly.
There had been some discussion about whether to base releases on features or fixed time schedules, or a combination thereof. I think a combination is probably the best, and would suggest the following:
X.Y releases (like 3.2 or 3.4) being released on a features basis. That is, when a sufficient number of the RFCs for the release are met, the Release Manager says its time to publish the new version. Hopefully this will happen on a fairly regular basis, but we all know how development can go...
X.Y.Z releases (like 3.2.1 or 3.2.2) being released on a fixed schedule. These updates include all the new features and bug fixes that the Quality Assurance Manager and Release Maintainer agree are stable and solid. There will be as many of these updates as there is time for before the next major release. I'm completely open to the periodicity of these releases.
As Chris points out, this is only possible if bug fixes and features can be isolated from one another for integration into the timed releases. Developers publishing early and often will help with this, as well as maintaining Git repos with separate branches for various developments. The new git.koha-community.org site has the potential to allow various development groups to push code to their own branches in a safe and secure way, so that may be one way for teams working on longer-term projects to more easily integrate separate features. I believe this could also improve collaboration across development teams.
Does this sound reasonable to folks? Am I missing any major points of consideration?
Cheers,
-Ian
On Wed, Jun 2, 2010 at 7:38 PM, Reed Wade <reed@catalyst.net.nz> wrote:
On Thu, Jun 3, 2010 at 3:07 AM, LAURENT Henri-Damien <laurenthdl@alinto.com> wrote:
Le 02/06/2010 12:22, Reed Wade a écrit :
Any reason not to move to a 4 week release cycle? With alternate ones getting "better" testing.
It would mean automating a lot of things -- but doable and worth it in any case.
^^The main reason is this one. and no doubt that it IS BOTH doable AND worth.... But in what time and who would do that... BibLibre is willing to devote time for that... But this surely is a community effort with development and should be managed as such. maybe there should be teams along with the assignees of bugs which could make smaller patches, and meetings so as to report progess.
(4 weeks is probably too short but...)
Yes, getting the science and organisation correct would be key.
Something ChrisC has alluded to in emails and IRC is the scheme we have developed at Catalyst for a project he and I and (5-8 others) work on. It is a long running development effort for a high scale news web site. It has a similar architecture and larger code base than Koha. We use git and there's a lot of perl. It seems likely there are some bits of code and techniques that could be applied to Koha release mgmt.
We have a 4 (really 1-6) week release cycle which is feature and date driven (customer might need new thing in time for national elections or sporting event and they don't tend to plan too far ahead).
We use debian packages for all code deployments and DB updates.
We use a system similar to bugzilla for tracking code development. Each feature or bug fix gets a branch named according to the ticket number.
We have an automated process which merges all branches for the next release (it queries the issue tracker to know what tickets are in what release) and emails the authors if there's a conflict. This runs several times a day. Instructions embedded in the ticket comments are used to sort dependencies and additional / alternate branches if needed for conflict resolution.
We have a similar automated process which performs the merge then creates debian packages for our testing environment. It emails the results to the release manage who reviews and sets the status of updated features so QA staff know what they can now test (this should be automated but isn't just yet because I like to review these still).
We have a pre-production autobuild set up which is a duplicate of all that but it only takes branches that QA have signed off on (examines ticket status).
This scheme has been extraordinarily satisfying and effective and allows us to spend more time on real work instead of the mechanics of the process. Production deployments used to be long bad days and now they tend to be uneventful. Everyone is happier. Release management is now time spent considering product features instead of the mechanics of integration and testing.
The key benefits we've gained are: - at any moment you can play with what would be the release if you decided to release at that moment - developers become responsible for fixing code conflicts or other blocks to their branch getting into a release and without hardly any time needed by the release manager (so very low latency for fixes) to wrangle that because it's automated emails telling the developer something went wrong
--
How to apply to Koha?
There are differences. We are a small group, mostly in the same office. Our customer is a 5 minute walk down the street from us.
I was about to say that our trade-offs and priorities are clear -- but that wasn't true before we had the system described above. It's allowed us and the customer to focus on keeping things clear because they've seen the value of things like having a feature freeze date that doesn't change unless the release date changes.
Some key things that could be taken--
- the auto-build and merge and report and package on a test machine for easy QA system -- this is something we should be able to adapt to bugzilla and make available for the project
- the notion of naming branches according to bug/feature id -- we've found this to be very powerful
Chris or Lars may have something to add.
-reed _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
-- Ian Walls Lead Development Specialist ByWater Solutions ALA Booth # 817 Phone # (888) 900-8944 http://bywatersolutions.com ian.walls@bywatersolutions.com Twitter: @sekjal
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
On 02/06/10 08:59, Paul Poulain wrote:
Hi,
In this mail, I want to share with you our situation, our problems, and our ideas. Thanks to Paul for triggering a needed discussion. The opportunities for Koha seem to be very big at the moment throughout the world and we are facing a lot of big decisions. I think the issues of more frequent releases is becoming more pressing. Increasingly we're all deploying 3.1. somethings or pre 3.2s or alphas because we need to respond to the needs of libraries more swiftly. Looking at perl's experience of having regular scheduled releases has convinced me that this is a good idea. The traditional feature driven release is driven more by marketing than user needs. Even if we can get regular unstable release points that might enables us to progress more constructively. I think we need to start thinking about technical and organizational issues for beyond 3.2 now. Possibly we should start working toward a 3.4 technical summit on irc where we could see if we can thrash out a plan.
Cheers Colin -- Colin Campbell Chief Software Engineer, PTFS Europe Limited Content Management and Library Solutions +44 (0) 208 366 1295 (phone) +44 (0) 7759 633626 (mobile) colin.campbell@ptfs-europe.com skype: colin_campbell2 http://www.ptfs-europe.com
Le 02/06/2010 10:25, Chris Cormack a écrit :
So as RM I accept the challenge of faster releases, and throw back a challenge of better patches :)
Chris & al, We (BibLibre & the community) face a big problem, and we must deal with it here and now : we (BibLibre) are deploying biblibre/master. It's live in Lyon3 university, it'll be live in Nimes next weeks, in SAN-OP in a few more weeks (migrating from 3.0). It contains a lot of new features (circ rules rewritten, and other new features described in wiki.koha-community.org, RFC for 3.4 from BibLibre) And we have a lot of new contracts falling. The technical fact : - we (BibLibre) have 450+ patches on git.biblibre.com, master branch, that are not in git.koha-community.org (new features & bugfixes) - koha-community has 100 patches that are not on biblibre/master So : if we do nothing, in a few *weeks*, we will have a fork that we want to avoid ! Here is our proposal : 1- release 3.2 2- we (BibLibre) rebase our master to 3.2 (that's 450+ patches) 3- we (BibLibre) submit them to master/koha 4- you (chris) push them into main trunk (as they are) 5- we (community) define the rules for all the next devs. 6- start working with new rules => september 1-Release 3.2 : ============= hdl will take care this next two weeks of the 2 BLO related to updatedatabase & affected to him 2 - rebase biblibre/master on koha/master ============== We (BibLibre) take care of rebasing our master at a date we define altogether. As that will be a HUGE work, you (chris) don't apply any other patches while we rebase to avoid breaking our work. We think we would need one week (maybe two) to rebase & test properly. 3/4 - push biblibre patches on koha/master ============== You (chris) push our 450+ patches on koha/master. Since the new rules were not defined when we wrote all this code, those patches don't abide by those rules and can't. 5 - define rules ============== Once 3.2 is released, we (community) set all rules (Including coding style, test cases, choosing OO pattern, standard commit notes,...) and everybody use them - including us (BibLibre) we don't request to be favored - 6 - work on 3.4 ============== All RFCs we wrote on the wiki are now implemented for our Nimes & Lyon 3 customers. If something new comes, we will respect fully the new rules. There may be some patches fixing Lyon3/Nimes features that won't respect them, but, hopefully, there will be only a few of them (Lyon3 is live, Nimes go live next week, so it's really stable) -- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc Tel : (33) 4 91 81 35 08
2010/6/22 Paul Poulain <paul.poulain@biblibre.com>:
Le 02/06/2010 10:25, Chris Cormack a écrit :
So as RM I accept the challenge of faster releases, and throw back a challenge of better patches :)
Chris & al,
We (BibLibre & the community) face a big problem, and we must deal with it here and now : we (BibLibre) are deploying biblibre/master. It's live in Lyon3 university, it'll be live in Nimes next weeks, in SAN-OP in a few more weeks (migrating from 3.0). It contains a lot of new features (circ rules rewritten, and other new features described in wiki.koha-community.org, RFC for 3.4 from BibLibre)
And we have a lot of new contracts falling.
The technical fact : - we (BibLibre) have 450+ patches on git.biblibre.com, master branch, that are not in git.koha-community.org (new features & bugfixes) - koha-community has 100 patches that are not on biblibre/master
So : if we do nothing, in a few *weeks*, we will have a fork that we want to avoid !
Here is our proposal : 1- release 3.2 2- we (BibLibre) rebase our master to 3.2 (that's 450+ patches) 3- we (BibLibre) submit them to master/koha 4- you (chris) push them into main trunk (as they are) 5- we (community) define the rules for all the next devs. 6- start working with new rules => september
1-Release 3.2 : ============= hdl will take care this next two weeks of the 2 BLO related to updatedatabase & affected to him
2 - rebase biblibre/master on koha/master ============== We (BibLibre) take care of rebasing our master at a date we define altogether. As that will be a HUGE work, you (chris) don't apply any other patches while we rebase to avoid breaking our work. We think we would need one week (maybe two) to rebase & test properly.
3/4 - push biblibre patches on koha/master ============== You (chris) push our 450+ patches on koha/master. Since the new rules were not defined when we wrote all this code, those patches don't abide by those rules and can't.
5 - define rules ============== Once 3.2 is released, we (community) set all rules (Including coding style, test cases, choosing OO pattern, standard commit notes,...) and everybody use them - including us (BibLibre) we don't request to be favored -
6 - work on 3.4 ============== All RFCs we wrote on the wiki are now implemented for our Nimes & Lyon 3 customers. If something new comes, we will respect fully the new rules. There may be some patches fixing Lyon3/Nimes features that won't respect them, but, hopefully, there will be only a few of them (Lyon3 is live, Nimes go live next week, so it's really stable) --
This is certainly a possible plan of action, but no, I won't be pushing any patches to master without them going through QA. Certainly not 450 in one go. They will need to be branched into smaller feature sets and each one tested and merged. What I am willing to do, is be less strict about accompanying tests as the patches were written before that, what I am not willing to do is merge 450 patches that haven't been looked at by QA or the Release Manager into master. These aren't new rules. Splitting them up into featuresets will make their integration much faster. Chris
On Mon, Jun 21, 2010 at 2:44 PM, Chris Cormack <chris@bigballofwax.co.nz>wrote:
This is certainly a possible plan of action, but no, I won't be pushing any patches to master without them going through QA. Certainly not 450 in one go.
I don't know a lot about the programming aspects - but I have to agree with Chris on this. The idea of 450 patches going in without testing seems like a bit of a nightmare waiting to happen. It would mean finding bugs (which I'm famous for) and not being able to track down what caused it as easily. I'd feel more comfortable as a tester/doc manager/user if I knew thing were being tested in smaller chunks along the way.
Chris _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
This is certainly a possible plan of action, but no, I won't be pushing any patches to master without them going through QA. Certainly not 450 in one go. They will need to be branched into smaller feature sets and each one tested and merged. What I am willing to do, is be less strict about accompanying tests as the patches were written before that, what I am not willing to do is merge 450 patches that haven't been looked at by QA or the Release Manager into master. These aren't new rules. 1- we did a lot of QA on them, and although there may be some remaining bugs, we are live with them on 1 site, very soon on a second one, and many more. 2- we proposed to submit those patches many months ago, but galen decision was : "feature freeze for 3.2". We had to go ahead. 3- we will deal with any problem that may arise. And if 3.4 has to be released in something like 6 months, that's enough, undoubtfully 4- splitting them in smaller feature set is a more than huge work. It's
Le 21/06/2010 20:44, Chris Cormack a écrit : probably almost impossible. So, even if your proposition is a good idea, I think it's not a possible way to go for us. You're reaching our (BibLibre) limit. So, if someone is volunteering for this more-than-huge job, then, fair, head to git.biblibre.com/master, it's here, available, all commits comments are in english. But you can't count on us to do that, even if I'm sorry, really. Friendly. -- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc Tel : (33) 4 91 81 35 08
On 22 June 2010 07:40, Paul Poulain <paul.poulain@biblibre.com> wrote:
This is certainly a possible plan of action, but no, I won't be pushing any patches to master without them going through QA. Certainly not 450 in one go. They will need to be branched into smaller feature sets and each one tested and merged. What I am willing to do, is be less strict about accompanying tests as the patches were written before that, what I am not willing to do is merge 450 patches that haven't been looked at by QA or the Release Manager into master. These aren't new rules. 1- we did a lot of QA on them, and although there may be some remaining bugs, we are live with them on 1 site, very soon on a second one, and many more. 2- we proposed to submit those patches many months ago, but galen decision was : "feature freeze for 3.2". We had to go ahead. 3- we will deal with any problem that may arise. And if 3.4 has to be released in something like 6 months, that's enough, undoubtfully 4- splitting them in smaller feature set is a more than huge work. It's
Le 21/06/2010 20:44, Chris Cormack a écrit : probably almost impossible.
So, even if your proposition is a good idea, I think it's not a possible way to go for us. You're reaching our (BibLibre) limit. So, if someone is volunteering for this more-than-huge job, then, fair, head to git.biblibre.com/master, it's here, available, all commits comments are in english. But you can't count on us to do that, even if I'm sorry, really.
Well we will have to come up with some solution, they have not been tested with current master (like you say, they are 100 patches behind) and they will undoubtedly introduce bugs, this is no reflection on Biblibre, everyones code contains bugs, it is almost a mathematical impossibility that 450 patches will merge bug free. I will push them up into a new/biblibre-patches branch, and from there work can be done to isolate, test and merge them in managable chunks. We simply can't merge 450 patches now and not expect to make our lives much much harder in the future. I'm willing to work on making smaller feature sets, it will help a lot if the commit messages relate to bugs in bugzilla, that will make making them much easier, if not I will be asking Biblibre lots of questions. Also friendly :) Chris
Le 21/06/2010 21:48, Chris Cormack a écrit : (answer written after a restfull night)
Well we will have to come up with some solution, yes, we have ! They have not been tested with current master (like you say, they are 100 patches behind) and they will undoubtedly introduce bugs, this is no reflection on Biblibre, everyones code contains bugs, it is almost a mathematical impossibility that 450 patches will merge bug free.
I agree with you (the next question being : is it easier to fix some bugs introduced by a so big merge, or to split our work into small pieces, test them,... not sure of the answer)
I will push them up into a new/biblibre-patches branch, and from there work can be done to isolate, test and merge them in managable chunks. We simply can't merge 450 patches now and not expect to make our lives much much harder in the future. I'm willing to work on making smaller feature sets, it will help a lot if the commit messages relate to bugs in bugzilla, that will make making them much easier, if not I will be asking Biblibre lots of questions.
There is a "good" news I forget to speak of yesterday (it was 10PM, & i'm very busy, so very tired, those days) : Most (if not all) commits are related to a BibLibre mantis entry. Features and bugfixes. The bad news : it's all in french. The good news : 1- it should help seeing what is related to what 2- we could open our mantis repository to some of you. Note it contains a lot of private informations (about data migrations, comments,...), but we have sub-projects for every curstomer & split migration / feature devs, so we should be able to open only what needs to be open. I'll speak of that with hdl asap (not today, i'm training Aix-Marseille Universities) -- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc Tel : (33) 4 91 81 35 08
Paul Poulain wrote:
Le 21/06/2010 21:48, Chris Cormack a écrit :
We simply can't merge 450 patches now and not expect to make our lives much much harder in the future. I'm willing to work on making smaller feature sets, it will help a lot if the commit messages relate to bugs in bugzilla, that will make making them much easier, if not I will be asking Biblibre lots of questions.
A small thing to flag up: software.coop is in a similar position. We have had a fork since LibLime pushed new features into HEAD which didn't work yet on platforms we were contracted to support. So, we had a similar fork-or-fail choice to BibLibre and reached the same unhappy decision. I've been trying to tidy+push changes and move us towards merging, but that work has slowed this year because something was up with email pull requests and I had some large urgent paid projects. It's also been complicated by conflicting changes from LibLime being submitted as part of Harley. And merging gets no easier as more time passes. I'm sure BibLibre will appreciate this: it sounds like it's hard enough for them merging a fork after some months - ours has existed almost two years or so now :-( It's a double whammy: it means we're not testing and fixing the next Koha release. If big players like BibLibre are also in a similar situation, how much of the Koha 3.2 slip has this caused?
There is a "good" news I forget to speak of yesterday (it was 10PM, & i'm very busy, so very tired, those days) : Most (if not all) commits are related to a BibLibre mantis entry. Features and bugfixes.
The bad news : it's all in french.
I don't care. Submit the French. Then either us franglais speakers can translate on demand, or people can use machine translation to get the gist of it. It's far better than having a cryptic "MT" reference to something most of us can't read.
The good news : 1- it should help seeing what is related to what 2- we could open our mantis repository to some of you. Note it contains a lot of private informations (about data migrations, comments,...), but we have sub-projects for every curstomer & split migration / feature devs, so we should be able to open only what needs to be open.
Aha! So BibLibre made the same mistake we did with private information polluting our koha development work! I'm sure I remember being flamed about how simple it was to avoid or clean that. It's not. Nevertheless, it's good to see BibLibre following in our footsteps. I hope we can find some better solution than either the unhappy forker doing all the clean-up, or the confusing competing-release approach of PTFS Harley which leaves the community doing all the work as well as answering questions from users about the new Koha "release". Regards, -- MJ Ray (slef) Webmaster and LMS developer at | software www.software.coop http://mjr.towers.org.uk | .... co IMO only: see http://mjr.towers.org.uk/email.html | .... op
On 22 June 2010 22:01, MJ Ray <mjr@phonecoop.coop> wrote:
Paul Poulain wrote:
Le 21/06/2010 21:48, Chris Cormack a écrit :
We simply can't merge 450 patches now and not expect to make our lives much much harder in the future. I'm willing to work on making smaller feature sets, it will help a lot if the commit messages relate to bugs in bugzilla, that will make making them much easier, if not I will be asking Biblibre lots of questions.
A small thing to flag up: software.coop is in a similar position. We have had a fork since LibLime pushed new features into HEAD which didn't work yet on platforms we were contracted to support. So, we had a similar fork-or-fail choice to BibLibre and reached the same unhappy decision.
I've been trying to tidy+push changes and move us towards merging, but that work has slowed this year because something was up with email pull requests and I had some large urgent paid projects. It's also been complicated by conflicting changes from LibLime being submitted as part of Harley. And merging gets no easier as more time passes. I'm sure BibLibre will appreciate this: it sounds like it's hard enough for them merging a fork after some months - ours has existed almost two years or so now :-(
It's a double whammy: it means we're not testing and fixing the next Koha release. If big players like BibLibre are also in a similar situation, how much of the Koha 3.2 slip has this caused?
There is a "good" news I forget to speak of yesterday (it was 10PM, & i'm very busy, so very tired, those days) : Most (if not all) commits are related to a BibLibre mantis entry. Features and bugfixes.
The bad news : it's all in french.
I don't care. Submit the French. Then either us franglais speakers can translate on demand, or people can use machine translation to get the gist of it. It's far better than having a cryptic "MT" reference to something most of us can't read.
The good news : 1- it should help seeing what is related to what 2- we could open our mantis repository to some of you. Note it contains a lot of private informations (about data migrations, comments,...), but we have sub-projects for every curstomer & split migration / feature devs, so we should be able to open only what needs to be open.
Aha! So BibLibre made the same mistake we did with private information polluting our koha development work! I'm sure I remember being flamed about how simple it was to avoid or clean that. It's not.
Mantis is their Bug tracker, so I dont think Paul was saying the private data is in the code, more that it is in their bug tracker. Which seems fine. But just in case they do mean in git. Here is a good tutorial on removing sensitive data. http://help.github.com/removing-sensitive-data/ It isn't actually that bad (no flame intended :)).
Nevertheless, it's good to see BibLibre following in our footsteps. I hope we can find some better solution than either the unhappy forker doing all the clean-up, or the confusing competing-release approach of PTFS Harley which leaves the community doing all the work as well as answering questions from users about the new Koha "release".
I don't think there is a need for a fork in this case, once the patches are rebased on master, and pushed into their own branch, we can divide them up, test and merge. But in the meantime once that branch is based on master, and rebased regularly, it's very simple for people to merge that into a branch in their own repository, until its all in master. Chris
Le 22/06/2010 12:09, Chris Cormack a écrit :
Aha! So BibLibre made the same mistake we did with private information polluting our koha development work! I'm sure I remember being flamed about how simple it was to avoid or clean that. It's not.
Mantis is their Bug tracker, so I dont think Paul was saying the private data is in the code, more that it is in their bug tracker. Which seems fine.
But just in case they do mean in git. Here is a good tutorial on removing sensitive data. http://help.github.com/removing-sensitive-data/ It isn't actually that bad (no flame intended :)).
chris is right : it's just our mantis that has some sensitive information. git is clean (as it is and has always been 100% available on git.biblibre.com) -- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc Tel : (33) 4 91 81 35 08
On Mon, Jun 21, 2010 at 3:40 PM, Paul Poulain <paul.poulain@biblibre.com> wrote:
Le 21/06/2010 20:44, Chris Cormack a écrit :
This is certainly a possible plan of action, but no, I won't be pushing any patches to master without them going through QA. Certainly not 450 in one go. They will need to be branched into smaller feature sets and each one tested and merged. What I am willing to do, is be less strict about accompanying tests as the patches were written before that, what I am not willing to do is merge 450 patches that haven't been looked at by QA or the Release Manager into master. These aren't new rules. 1- we did a lot of QA on them, and although there may be some remaining bugs, we are live with them on 1 site, very soon on a second one, and many more.
But this approach is not acceptable in the community at large as evidenced by the BibLibre acquisitions work introducing four blockers. What you and your clients may be able to live with, others may not be able to live with. So not only do we need good QA at the vendor level, but also at the community level as well.
2- we proposed to submit those patches many months ago, but galen decision was : "feature freeze for 3.2". We had to go ahead.
This had to happen some time. I would suggest that the problem is not with the feature freeze, but perhaps with not keeping in sync with the main repo master.
3- we will deal with any problem that may arise. And if 3.4 has to be released in something like 6 months, that's enough, undoubtfully
I think there is a better way. One which requires less resources on the part of one company. I cannot imagine agreeing to take on every possible problem that may arise from the merging of 400+ patches into a branch that is already 100 patches behind. I would not expect any company to take that upon themselves when we can do it in a much cleaner manner.
4- splitting them in smaller feature set is a more than huge work. It's probably almost impossible.
I think that at a minimum the present BibLIbre work would need to be merged into a topic branch off of the stable 3.2 master and then testing/debugging be done on that topic branch while keeping it in sync with the stable master. In no case does it appear that applying 400+ patches to a stable master would be possible. *That* would make for huge work.
So, even if your proposition is a good idea, I think it's not a possible way to go for us. You're reaching our (BibLibre) limit. So, if someone is volunteering for this more-than-huge job, then, fair, head to git.biblibre.com/master, it's here, available, all commits comments are in english. But you can't count on us to do that, even if I'm sorry, really.
My only observation here would be that the burden of keeping development work in sync with the official HEAD *must* always fall primarily on the developer and not the community. This is a relatively easy task seeing we are using git version control and not svn or some other less friendly sort. Again, I think the new features are great. Its the logistics that are the fly in the ointment so to speak. Kind Regards, Chris
Le 21/06/2010 22:04, Chris Nighswonger a écrit :
On Mon, Jun 21, 2010 at 3:40 PM, Paul Poulain <paul.poulain@biblibre.com> wrote:
Le 21/06/2010 20:44, Chris Cormack a écrit :
This is certainly a possible plan of action, but no, I won't be pushing any patches to master without them going through QA. Certainly not 450 in one go. They will need to be branched into smaller feature sets and each one tested and merged. What I am willing to do, is be less strict about accompanying tests as the patches were written before that, what I am not willing to do is merge 450 patches that haven't been looked at by QA or the Release Manager into master. These aren't new rules.
1- we did a lot of QA on them, and although there may be some remaining bugs, we are live with them on 1 site, very soon on a second one, and many more.
But this approach is not acceptable in the community at large as evidenced by the BibLibre acquisitions work introducing four blockers.
I never wrote that publicly, but I must say i'm not very proud of the new acq code. it can/must be improved a lot.
What you and your clients may be able to live with, others may not be able to live with. So not only do we need good QA at the vendor level, but also at the community level as well.
of course. the idea being that merging very early (reminder : everything is written) means we would have 6 months to find & fix bugs. It's not the same as merging 10 days before releasing imo.
2- we proposed to submit those patches many months ago, but galen decision was : "feature freeze for 3.2". We had to go ahead.
This had to happen some time.
agreed. The problem being that 3.2 is very late. Today, we have to do something to merge our work.
I would suggest that the problem is not with the feature freeze, but perhaps with not keeping in sync with the main repo master.
we tried, but had some bugs introduced by the merge + we were too short on time. Again : today, we (BibLibre) are willing to do something because in a few weeks, it'll be too late, unfortunately
I think that at a minimum the present BibLIbre work would need to be merged into a topic branch off of the stable 3.2 master and then testing/debugging be done on that topic branch while keeping it in sync with the stable master.
see answer to the other chris for the rest of your mail ;-) -- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc Tel : (33) 4 91 81 35 08
Paul Poulain skribis:
Le 21/06/2010 22:04, Chris Nighswonger a écrit :
On Mon, Jun 21, 2010 at 3:40 PM, Paul Poulain <paul.poulain@biblibre.com> wrote:
2- we proposed to submit those patches many months ago, but galen decision was : "feature freeze for 3.2". We had to go ahead.
This had to happen some time.
agreed. The problem being that 3.2 is very late. Today, we have to do something to merge our work.
OK, so problems in 3.0's release process caused software.coop to fork; and now problems in 3.2's release process caused BibLibre to fork. Can we do anything to stop 3.4 causing another vendor to fork?
I would suggest that the problem is not with the feature freeze, but perhaps with not keeping in sync with the main repo master.
we tried, but had some bugs introduced by the merge + we were too short on time.
Just a side note: I think the co-op fork has most patches from master integrated, but it's still non-trivial to clean and push back.
Again : today, we (BibLibre) are willing to do something because in a few weeks, it'll be too late, unfortunately
Great, so if giving it a free pass through community QA isn't an option, what's the next best choice? Regards, -- MJ Ray (slef) Webmaster and LMS developer at | software www.software.coop http://mjr.towers.org.uk | .... co IMO only: see http://mjr.towers.org.uk/email.html | .... op
Hi Paul, 2010/6/21 Paul Poulain <paul.poulain@biblibre.com>:
The technical fact : - we (BibLibre) have 450+ patches on git.biblibre.com, master branch, that are not in git.koha-community.org (new features & bugfixes) - koha-community has 100 patches that are not on biblibre/master
Ouch!
So : if we do nothing, in a few *weeks*, we will have a fork that we want to avoid !
Great!
1-Release 3.2 : ============= hdl will take care this next two weeks of the 2 BLO related to updatedatabase & affected to him
2 - rebase biblibre/master on koha/master ============== We (BibLibre) take care of rebasing our master at a date we define altogether. As that will be a HUGE work, you (chris) don't apply any other patches while we rebase to avoid breaking our work. We think we would need one week (maybe two) to rebase & test properly.
I think, this would be the place to implement the first of Chris' suggestions: Have hdl (or whoever) do this merging in such a way as to produce smaller feature-centric branches. (Think "bite sized" pieces.) However, I think this must be done concurrent to development on 3.4. We must also consider that if we are ever to have schedule based releases, we *must* take this opportunity to hold on to a stable master and use branching along with merging to facilitate ongoing development while maintaining that stable master. So here is a good place to begin. As BibLibre finishes up merging on a particular feature branch, QA can immediately pickup with testing and then the RM take care of merging the tested branch into master, thus keeping master stable. I think if everyone pitches in that this could all happen in a month time frame. As you say, these new features are in production on several of your large clients, so they should not be too buggy.
3/4 - push biblibre patches on koha/master ============== You (chris) push our 450+ patches on koha/master. Since the new rules were not defined when we wrote all this code, those patches don't abide by those rules and can't.
As Chris and Nicole have pointed out, I don't think a direct push into a stable master of this large of a patch set will prove beneficial to anyone.
5 - define rules ============== Once 3.2 is released, we (community) set all rules (Including coding style, test cases, choosing OO pattern, standard commit notes,...) and everybody use them - including us (BibLibre) we don't request to be favored -
6 - work on 3.4 ============== All RFCs we wrote on the wiki are now implemented for our Nimes & Lyon 3 customers. If something new comes, we will respect fully the new rules. There may be some patches fixing Lyon3/Nimes features that won't respect them, but, hopefully, there will be only a few of them (Lyon3 is live, Nimes go live next week, so it's really stable)
The rest of this naturally follows along of course. We need some enforcement of rules especially as the project gets bigger and larger support companies are pushing the development at a greater pace. It will be especially imperative that those who have the resources to push development faster adhere more strictly to good coding/testing/etc practices as messy code in large quantities will quickly produce chaos and be of little use in the long haul. Kudos to BbLibre for being open and transparent in all of this and for requesting community input as they undergo growing pains. Kind Regards, Chris
As everyone has pointed out 400+ is a lot and liable to generate a bout of indigestion, but as Paul pointed out in his original email we need to up our game when it comes to managing the product of all that work. Small improvements could make life easier for all. If the patch introduces a feature or changes how one behaves is it documented? The pod should answer the puzzled why of anyone reading the code. Are there tests? (And if so does it pass them) We really need to take testing much more seriously, coverage of the code is minimal, (and we still don't have a code base that runs cleanly with use warnings!!). There are some trivial things you could do, before submitting the patch make sure that you're not adding any meaningless whitespace at the end of lines ( git diff shows it) it helps keep future diffs, merges etc less labour-intensive. Does your code show its intention clearly. Trivial points like variable names (data and results don't convey much more than using random strings of hex digits), don't be obscure (changing e.g. hldct to hold_count removes ambiguity especially to a fellow programmer who may not be a native speaker of your dialect of English). Does the code need breaking up into subroutines? (Programmers have very short attention spans - about a screen and a bit and even shorter in if else blocks). Colin -- Colin Campbell Chief Software Engineer, PTFS Europe Limited Content Management and Library Solutions +44 (0) 208 366 1295 (phone) +44 (0) 7759 633626 (mobile) colin.campbell@ptfs-europe.com skype: colin_campbell2 http://www.ptfs-europe.com
Le 22/06/2010 23:27, Reed Wade a écrit :
400 patches
Chris and I needed to do some checking before making a public commitment but--
Catalyst will be able supply a couple of weeks of someone from our QA team to help resolve these.
That would be awesome ! you can count on us to work closely with you on this matter ! Hurray for catalyst ! -- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc Tel : (33) 4 91 81 35 08
participants (11)
-
Chris Cormack -
Chris Nighswonger -
Colin Campbell -
Ian Walls -
Lars Wirzenius -
LAURENT Henri-Damien -
LAURENT Henri-Damien -
MJ Ray -
Nicole Engard -
Paul Poulain -
Reed Wade