[Koha-devel] BibLibre strategy for Koha 3.4 and next versions

[Chris Cormack]

Hi Paul

There is an IRC meeting today, in 3 hours time .. but this topic is
much to big for that, so I propose we leave it off the agenda for
that, and let people think about the ideas raised for a bit, and then
do replies.

Speaking for myself nothing in the email strikes me as bad news, I do
think time based releases cause their own set of problems, and as  RM
elect I think if we decided on that we would certainly have to become
a LOT stricter about patches. IE, we would need to reject more and
faster .. which might not be a bad thing anyway.
I also don't think bug wrangler is a small job .. .anyone who thinks
that hasnt looked in bugzilla recently ;-) Some of the main tasks of a
wrangler are just what you describe Henri-Damien as doing, ie,
managing integration of bug fixes and new features.

One thing that would allow patches to be accepted a lot faster would
be making every patch need to contain a working test. One thing I am
considering is borrowing from the Java people and getting Hudson up
and running http://wiki.hudson-ci.org/display/HUDSON/Meet+Hudson
Of course, we need unit tests to make this useful.

As for the technical part of managing many branches in git, we have
significant experience of that at work, in the team I work on every
single feature, or bug, is in its' own branch and they are merged into
testing releases (we have literally hundreds of branches). Git makes
this easy, so I think the use of many more feature branches at
git.koha-community.org will make merging a lot easier, this means even
virtual forks are only just a branch.

As to the speed of pushing patches, test cases and good commit
messages would improve that immensely. But I am confident that with
the help of a QA manager, which poor Galen never had, we can accept or
reject patches in a timely manner. The better your commit message the
easier that gets, if we have to read all the code just to try and
figure out what its trying to do, then to figure out if it is actually
doing it .. that will slow things down immensely.

So as RM I accept the challenge of faster releases, and throw back a
challenge of better patches :)

Chris

2010/6/2 Paul Poulain <paul.poulain at biblibre.com>:
> Hi,
>
> In this mail, I want to share with you our situation, our problems, and our
> ideas.
> You'll be probably surprised by this mail. Please consider it's a good news.
> It's not a threat - not my cup of tea (frenchism suspected) - . But I think
> things must be done this way, so I share it with you, and I'm impatiently
> waiting for koha-community reaction & thoughts.
>
> BibLibre situation
> ===========
> We are very successful. Koha is really a success in France. One french
> university is now live, we are on the way to migrate a very large public
> library (500k items), 3 other universities should go live in September, and
> a lot of business coming.
> We are 11, are on our way to hire 3 new ppl, have on board Philippe
> Chabanon, that is very experienced in managing a company, is fully
> enthusiastic about free softwares and community, and knows a lot of people
> in the french library world.
> Is everything perfect ? nope.
>
> Our problems
> =========
> Organizational problem
> ------------
> Some of our clients sponsor new developments. Those are done under "marché
> public" rules, that usually define a very strict date (if we fail, then we
> have to pay fines, based on how many days late we are !). That's why we
> started 3 different branches for our 3 different major sponsored works
> (Aix-Marseille, Nîmes, Lyon 3, you can see everything on git.biblibre.com).
> Everything that was in Aix-Marseille branch is now in koha 3.2 . But we have
> a lot of things done for Lyon 3 (that is live already) and Nimes (that will
> go live in early july) that are not in koha 3.2
>
> Our main short-term problem is that koha 3.2 is very (very) late. It has
> been announced one year ago ! We did a lot of testing for Lyon 3 before
> putting them live. Some of them resulted in some bugfixes that could/should
> have been submitted to koha mainstream. But, we couldn't. Thus we face a
> situation where someone fixes in official 3.2 some things we already fixed
> in our Lyon3 branch (shame on us, but really, we couldn't find time.
> Delivery to Lyon3 was on time, but is was shorter than short)
> The situation is the same with Nimes : they will probably be live (with 3.2
> + Lyon3 + Nimes sponsored stuff) before 3.2 is released -going live planned
> for july 1st, Galen, we challenge you ;-) -.
>
> So, frankly, dealing with our contracts and the official version of Koha has
> been a pain, and has resulted in a "virtual fork" for Lyon3 (and in 1 month
> for Nimes). We strongly want to submit our code to the main trunk, but
> dealing with a so-long-delayed version is really a nightmare (and I
> understand PTFS deploying "Harley" -even if i share community opinion about
> the version being advertised when released in a way it can be understood as
> an official version-). We must find a solution to this situation, or the
> "virtual fork" will result in a "real fork" that we strongly want to avoid !
>
> As of today, we have something like 400 patches to integrate into koha main
> trunk !
>
> Technical problem
> -----------
> With some customers having more than 1 million items (Aix-Marseille :
> 1247686), and with more and more customers, we reach some limits with the
> structure of Koha, in term of performances and stability/maintainability.
> Chris suggested some improvements with koha 3.4, and we have some
> complementary ideas, I want to share with you, and would like to implement
> ASAP.
>
> Ideas/proposal
> =======
> Technical : Koha architecture
> ---------------------------------
> At the begining of may 2010, we staff had a meeting at Marseille to know if
> it would be possible to make koha easier to maintain and evolve, we are now
> doing some experiment to split koha into several layers that provides
> completely independant set of tasks based on the MVC architecture
> (http://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller).
>
> Koha can be seen as a stack with communication between the closest layers. 4
> layers at all (does it make you remember TCP/IP? ;) )
>
> VIEW: interactions with different clients ( browser, REST, XMPP, Z3950, ...
> )
> ↓↑
> CONTROLLER: Controls the model by transmitting the CRUD tasks requested by
> the VIEW
> ↓↑
> MODEL: Business objects, as close as possible to the real world
> ↓↑
> PERSISTENCE: Make the Business objects persistent
>
> Plus, we expect to use external tools as often as possible to take
> experience of other communities and reduce the amount of code we have to
> debug by ourselves. We think tools exists for every layers, actually we just
> have to
> choose between them:
>
> View:
> -------
> HTML::Template, Template Toolkit 2, Template::Declare, SOAP::Lite,
> any REST, XMPP, Z3950, SRU/SRW wrapper, ... (Infobot ;))
>
> Controller:
> ----------
> Dancer, Mojo, The Controller parts of existing MVC frameworks (
> Jifty, Catalyst, ...), ...
>
> Model:
> -------
> Any OO Framework ( prefer those which comes with persistence extension as
> MooseX::Storage, KiokuDB, ... choose one )
>
> Persistence:
> -----------
> DBIx::Class, Document DB clients, SQL::Abstract, Jifty::DBI, ....
>
> Ideally, the communication between each layers must be specified, provable
> by unit testing, and well documented ( pods in the t/* ?).
>
> Today, Koha works in CGI mode, so there is no persistence at all. And having
> no persistence means we can't implement the MVC model -some tests we did
> with OO frameworks show very poor performances-, and won't be able to
> improve performance heavily -the main loss of perf in koha as of today is
> loading datas that are almost static & should be available immediatly :
> systempreferences & all the admin tables -.
>
> So: 1st step => remove CGI, to have persistent code, enabling persistent DB
> connection. We did some tests with Dancer, that are very encouraring, we
> will send a mail about that soon.
> 2nd step => Once we will be OK with dancer to get rid of CGI => head to
> persistence & OO framework. We vote for Moose & KiokuDB, but we still have
> POC to write to confirm the idea.
> 3rd step =>view part, by moving to T::T (POC still to do to confirm the
> idea)
>
> It may look like a koha rewrite but it's not: every parts would be added
> step by step into koha by wrapping the new design behind the old API during
> a transition phase. An example of current code
>
> sub GetBranchesInCategory($) {
> my ($categorycode) = @_;
> my @branches;
> my $dbh = C4::Context->dbh();
> my $sth = $dbh->prepare(
> "SELECT b.branchcode
> FROM branchrelations r, branches b
> where r.branchcode=b.branchcode and r.categorycode=?"
> );
> $sth->execute($categorycode);
> while ( my $branch = $sth->fetchrow ) {
> push @branches, $branch;
> }
> $sth->finish();
> return ( \@branches );
> }
>
> would became transition code:
>
> sub GetBranchesInCategory($) {
> my $id = shift
> or return;
>
> # - Category would be the Business object with a member transition (which
> can be
> # deprecated at the end of transition) that encapsulate every
> # persistance specific data used in the C4::* api
> # - the find method could be delegated to the DBIx::Class find that returns
> the
> # record associated to the id
>
> # returns
> if ( my $category = Category->transition->find($id) ) {
> # codes to deal with branches comes de facto with tools as DBIx::Class
> # it returns the list of branches for the category
> $category->branches;
> }
> else { undef }
> }
>
> You can find in attachement a small graph with all our ideas, and the
> beginning of a timeline.
>
> ppl reading carefully will see a "changing indexing system (solr)". 4 years
> after starting with it, we still find zebra being a nightmare, with a lot of
> problems for unicode (with ICU). We did some tests with solr, and they are
> very encouraging. We will provide the results soon too. But i'm really
> enthusiast about what we did in a few hours !
>
> Organizational : Koha team
> -------------------------------
>     Koha team organisation
>     ...............
> If we (BibLibre & the Koha community) want to be able to implement such a
> roadmap (that also includes some new functionnality, as customers sponsors
> some...), we must have a heavily organised team. I think we must :
> * manage release by date (not by features). This is probably the more
> important part for us (see our "marché public" problem). The timing of the
> versions would be defined by the companies contributing the most and synched
> with sponsored devs roadmap. That will be tricky to organize when there are
> more than one company involved, but, for instance, BibLibre is the largest
> contributor ( PTFS: you're welcomed to enter the thread ! ). In fact, "a
> koha 3.4" could be released in a few weeks (look at out master branch, it
> contains 3.2 + Lyon3 + Nimes sponsored work). A "koha 3.6" could be released
> in december/february,... a regular release, that's something that is
> probably "non-negociable" for us.
> * define very clear coding guidelines & be very stricts on their use :
> everything that does not respect the guidelines must be politely but firmly
> rejected.
> * have a very efficient patch workflow : a patch can't be in the wild for
> weeks. The submitter must have an answer in a very short timeline (is 48H
> possible ?). ie : We can't rely on only one person to push patches. We must
> have back-ups for major roles like RM & "patch pusher".
> * we could divide Koha in functionnal (serials / acquisition / cataloguing /
> ...) or technical (perfs / SQL structure / view layer / indexing engine /
> ...) parts, with some working group for each of them, and something like a
> monthly report of the WG (with a weekly internal meeting)
> * write tests ! technical tests (.t) and functional tests (selenium ?)
>
>     Our involvement (BibLibre) :
>     ....................
> We want to improve our involvement in Koha, by dedicating Henri-Damien to
> work at 75% for Koha and be the link between the work done internally by
> BibLibre, and the work done by everyone else. This mean :
>
> publish our RFPs as soon as possible
> gather all the work done every month and communicate about that on the wiki
> and blog and translate on koha-fr.org
> manage patches coming out from BibLibre, and coming out from everyone else
> (to deal as soon as possible with any problem)
>
> I know we have elected a team recently, and I know chris will do an awesome
> job as RM (I don't know for others, I didn't saw them in their
> responsability yet. For chris, i'm the team for 8 years, I know who he is
> ;-) ), but I think hdl should have a responsability more important than "bug
> wrangler" to reflect the time he (will) dedicate to the project. Is
> something like "co-RM" possible ? any other idea ? Any other idea ?
>
> Next step
> -----------
> We are *really* willing to implement this roadmap and way of doing things.
> The goal of this mail is to ask everybody involved in Koha development about
> their opinion: we want our proposal to be a chance for Koha (new deal ?).
>
> On the technical side, stay tuned. Some POC will fall in the next weeks !!!
> On the organizational side, the ball is on your side: let's go to
> update/improve our organization !
>
> --
> Paul POULAIN
> http://www.biblibre.com
> Expert en Logiciels Libres pour l'info-doc
> Tel : (33) 4 91 81 35 08
>
> _______________________________________________
> Koha-devel mailing list
> Koha-devel at lists.koha-community.org
> http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
>