Le 02/06/2010 10:25, Chris Cormack a écrit :
Hi Paul
There is an IRC meeting today, in 3 hours time .. but this topic is much to big for that, so I propose we leave it off the agenda for that, and let people think about the ideas raised for a bit, and then do replies.
Hi chris, this mail was sent today so that people can react, the intent was not to debate of that during today's meeting (agenda is already online) but to raise our questions and tell our point of view. Anyway, decisions and actions will have to be taken forward.
Speaking for myself nothing in the email strikes me as bad news, I do think time based releases cause their own set of problems, and as RM elect I think if we decided on that we would certainly have to become a LOT stricter about patches. IE, we would need to reject more and faster .. which might not be a bad thing anyway. I also don't think bug wrangler is a small job .. .anyone who thinks that hasnt looked in bugzilla recently ;-) Some of the main tasks of a wrangler are just what you describe Henri-Damien as doing, ie, managing integration of bug fixes and new features.
One thing that would allow patches to be accepted a lot faster would be making every patch need to contain a working test. One thing I am considering is borrowing from the Java people and getting Hudson up and running http://wiki.hudson-ci.org/display/HUDSON/Meet+Hudson Of course, we need unit tests to make this useful.
As for the technical part of managing many branches in git, we have significant experience of that at work, in the team I work on every single feature, or bug, is in its' own branch and they are merged into testing releases (we have literally hundreds of branches). Git makes this easy, so I think the use of many more feature branches at git.koha-community.org will make merging a lot easier, this means even virtual forks are only just a branch.
The way koha code is now is making that troublesome since when we fix some circulation stuff or holding stuff, we have to touch many files, so that when rebasing, we have many conflicts. Moreover, since our customers ask for features and not only bug fixes, and want a kind of relesase when they are delivered, we have to merge both devs and bugfixes in order to achieve global stability and coherence betweeen devs and bug fixes. Some devs we did also helped us in fixing bugs quicker, simpler and better. So that if we ported that to master back, on order to ensure that nothing is broken, i need more than just a few hours, would require me not just cherry-picking but also some kind of refactoring and then would introduce some risks both in koha-community master branch (overlooked bug) and in our branch when we rebase on koha-community master branch. This is why we also propose some new tools in order to improve coding guidelines and eventually have some way to automate testing.
As to the speed of pushing patches, test cases and good commit messages would improve that immensely. But I am confident that with the help of a QA manager, which poor Galen never had, we can accept or reject patches in a timely manner. The better your commit message the easier that gets, if we have to read all the code just to try and figure out what its trying to do, then to figure out if it is actually doing it .. that will slow things down immensely.
So as RM I accept the challenge of faster releases, and throw back a challenge of better patches :)
Well, there are some good exemples, and we are getting better, but it is also quite difficult to raise the cursor to the good point and when we do a quick followup, difficult to resist the temptation of using Followup.... And when we are having six months delay on integration of our branch, it is difficult to re process the whole history. It is the work of release manager, QA manager, Bug wrangler, and eventually, all the subscribers to koha-patches to ask for details. But what about branches to come ? quickly sent... my 2 cents. -- Henri-Damien LAURENT
Chris
2010/6/2 Paul Poulain <paul.poulain@biblibre.com>:
Hi,
In this mail, I want to share with you our situation, our problems, and our ideas. You'll be probably surprised by this mail. Please consider it's a good news. It's not a threat - not my cup of tea (frenchism suspected) - . But I think things must be done this way, so I share it with you, and I'm impatiently waiting for koha-community reaction & thoughts.
BibLibre situation =========== We are very successful. Koha is really a success in France. One french university is now live, we are on the way to migrate a very large public library (500k items), 3 other universities should go live in September, and a lot of business coming. We are 11, are on our way to hire 3 new ppl, have on board Philippe Chabanon, that is very experienced in managing a company, is fully enthusiastic about free softwares and community, and knows a lot of people in the french library world. Is everything perfect ? nope.
Our problems ========= Organizational problem ------------ Some of our clients sponsor new developments. Those are done under "marché public" rules, that usually define a very strict date (if we fail, then we have to pay fines, based on how many days late we are !). That's why we started 3 different branches for our 3 different major sponsored works (Aix-Marseille, Nîmes, Lyon 3, you can see everything on git.biblibre.com). Everything that was in Aix-Marseille branch is now in koha 3.2 . But we have a lot of things done for Lyon 3 (that is live already) and Nimes (that will go live in early july) that are not in koha 3.2
Our main short-term problem is that koha 3.2 is very (very) late. It has been announced one year ago ! We did a lot of testing for Lyon 3 before putting them live. Some of them resulted in some bugfixes that could/should have been submitted to koha mainstream. But, we couldn't. Thus we face a situation where someone fixes in official 3.2 some things we already fixed in our Lyon3 branch (shame on us, but really, we couldn't find time. Delivery to Lyon3 was on time, but is was shorter than short) The situation is the same with Nimes : they will probably be live (with 3.2 + Lyon3 + Nimes sponsored stuff) before 3.2 is released -going live planned for july 1st, Galen, we challenge you ;-) -.
So, frankly, dealing with our contracts and the official version of Koha has been a pain, and has resulted in a "virtual fork" for Lyon3 (and in 1 month for Nimes). We strongly want to submit our code to the main trunk, but dealing with a so-long-delayed version is really a nightmare (and I understand PTFS deploying "Harley" -even if i share community opinion about the version being advertised when released in a way it can be understood as an official version-). We must find a solution to this situation, or the "virtual fork" will result in a "real fork" that we strongly want to avoid !
As of today, we have something like 400 patches to integrate into koha main trunk !
Technical problem ----------- With some customers having more than 1 million items (Aix-Marseille : 1247686), and with more and more customers, we reach some limits with the structure of Koha, in term of performances and stability/maintainability. Chris suggested some improvements with koha 3.4, and we have some complementary ideas, I want to share with you, and would like to implement ASAP.
Ideas/proposal ======= Technical : Koha architecture --------------------------------- At the begining of may 2010, we staff had a meeting at Marseille to know if it would be possible to make koha easier to maintain and evolve, we are now doing some experiment to split koha into several layers that provides completely independant set of tasks based on the MVC architecture (http://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller).
Koha can be seen as a stack with communication between the closest layers. 4 layers at all (does it make you remember TCP/IP? ;) )
VIEW: interactions with different clients ( browser, REST, XMPP, Z3950, ... ) ↓↑ CONTROLLER: Controls the model by transmitting the CRUD tasks requested by the VIEW ↓↑ MODEL: Business objects, as close as possible to the real world ↓↑ PERSISTENCE: Make the Business objects persistent
Plus, we expect to use external tools as often as possible to take experience of other communities and reduce the amount of code we have to debug by ourselves. We think tools exists for every layers, actually we just have to choose between them:
View: ------- HTML::Template, Template Toolkit 2, Template::Declare, SOAP::Lite, any REST, XMPP, Z3950, SRU/SRW wrapper, ... (Infobot ;))
Controller: ---------- Dancer, Mojo, The Controller parts of existing MVC frameworks ( Jifty, Catalyst, ...), ...
Model: ------- Any OO Framework ( prefer those which comes with persistence extension as MooseX::Storage, KiokuDB, ... choose one )
Persistence: ----------- DBIx::Class, Document DB clients, SQL::Abstract, Jifty::DBI, ....
Ideally, the communication between each layers must be specified, provable by unit testing, and well documented ( pods in the t/* ?).
Today, Koha works in CGI mode, so there is no persistence at all. And having no persistence means we can't implement the MVC model -some tests we did with OO frameworks show very poor performances-, and won't be able to improve performance heavily -the main loss of perf in koha as of today is loading datas that are almost static & should be available immediatly : systempreferences & all the admin tables -.
So: 1st step => remove CGI, to have persistent code, enabling persistent DB connection. We did some tests with Dancer, that are very encouraring, we will send a mail about that soon. 2nd step => Once we will be OK with dancer to get rid of CGI => head to persistence & OO framework. We vote for Moose & KiokuDB, but we still have POC to write to confirm the idea. 3rd step =>view part, by moving to T::T (POC still to do to confirm the idea)
It may look like a koha rewrite but it's not: every parts would be added step by step into koha by wrapping the new design behind the old API during a transition phase. An example of current code
sub GetBranchesInCategory($) { my ($categorycode) = @_; my @branches; my $dbh = C4::Context->dbh(); my $sth = $dbh->prepare( "SELECT b.branchcode FROM branchrelations r, branches b where r.branchcode=b.branchcode and r.categorycode=?" ); $sth->execute($categorycode); while ( my $branch = $sth->fetchrow ) { push @branches, $branch; } $sth->finish(); return ( \@branches ); }
would became transition code:
sub GetBranchesInCategory($) { my $id = shift or return;
# - Category would be the Business object with a member transition (which can be # deprecated at the end of transition) that encapsulate every # persistance specific data used in the C4::* api # - the find method could be delegated to the DBIx::Class find that returns the # record associated to the id
# returns if ( my $category = Category->transition->find($id) ) { # codes to deal with branches comes de facto with tools as DBIx::Class # it returns the list of branches for the category $category->branches; } else { undef } }
You can find in attachement a small graph with all our ideas, and the beginning of a timeline.
ppl reading carefully will see a "changing indexing system (solr)". 4 years after starting with it, we still find zebra being a nightmare, with a lot of problems for unicode (with ICU). We did some tests with solr, and they are very encouraging. We will provide the results soon too. But i'm really enthusiast about what we did in a few hours !
Organizational : Koha team ------------------------------- Koha team organisation ............... If we (BibLibre & the Koha community) want to be able to implement such a roadmap (that also includes some new functionnality, as customers sponsors some...), we must have a heavily organised team. I think we must : * manage release by date (not by features). This is probably the more important part for us (see our "marché public" problem). The timing of the versions would be defined by the companies contributing the most and synched with sponsored devs roadmap. That will be tricky to organize when there are more than one company involved, but, for instance, BibLibre is the largest contributor ( PTFS: you're welcomed to enter the thread ! ). In fact, "a koha 3.4" could be released in a few weeks (look at out master branch, it contains 3.2 + Lyon3 + Nimes sponsored work). A "koha 3.6" could be released in december/february,... a regular release, that's something that is probably "non-negociable" for us. * define very clear coding guidelines & be very stricts on their use : everything that does not respect the guidelines must be politely but firmly rejected. * have a very efficient patch workflow : a patch can't be in the wild for weeks. The submitter must have an answer in a very short timeline (is 48H possible ?). ie : We can't rely on only one person to push patches. We must have back-ups for major roles like RM & "patch pusher". * we could divide Koha in functionnal (serials / acquisition / cataloguing / ...) or technical (perfs / SQL structure / view layer / indexing engine / ...) parts, with some working group for each of them, and something like a monthly report of the WG (with a weekly internal meeting) * write tests ! technical tests (.t) and functional tests (selenium ?)
Our involvement (BibLibre) : .................... We want to improve our involvement in Koha, by dedicating Henri-Damien to work at 75% for Koha and be the link between the work done internally by BibLibre, and the work done by everyone else. This mean :
publish our RFPs as soon as possible gather all the work done every month and communicate about that on the wiki and blog and translate on koha-fr.org manage patches coming out from BibLibre, and coming out from everyone else (to deal as soon as possible with any problem)
I know we have elected a team recently, and I know chris will do an awesome job as RM (I don't know for others, I didn't saw them in their responsability yet. For chris, i'm the team for 8 years, I know who he is ;-) ), but I think hdl should have a responsability more important than "bug wrangler" to reflect the time he (will) dedicate to the project. Is something like "co-RM" possible ? any other idea ? Any other idea ?
Next step ----------- We are *really* willing to implement this roadmap and way of doing things. The goal of this mail is to ask everybody involved in Koha development about their opinion: we want our proposal to be a chance for Koha (new deal ?).
On the technical side, stay tuned. Some POC will fall in the next weeks !!! On the organizational side, the ball is on your side: let's go to update/improve our organization !
-- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc Tel : (33) 4 91 81 35 08
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel