Hi,

In this mail, I want to share with you our situation, our problems, and our ideas.
You'll be probably surprised by this mail. Please consider it's a good news. It's not a threat - not my cup of tea (frenchism suspected) - . But I think things must be done this way, so I share it with you, and I'm impatiently waiting for koha-community reaction & thoughts.

BibLibre situation
===========
We are very successful. Koha is really a success in France. One french university is now live, we are on the way to migrate a very large public library (500k items), 3 other universities should go live in September, and a lot of business coming.
We are 11, are on our way to hire 3 new ppl, have on board Philippe Chabanon, that is very experienced in managing a company, is fully enthusiastic about free softwares and community, and knows a lot of people in the french library world.
Is everything perfect ? nope.

Our problems
=========
Organizational problem
------------
Some of our clients sponsor new developments. Those are done under "marché public" rules, that usually define a very strict date (if we fail, then we have to pay fines, based on how many days late we are !). That's why we started 3 different branches for our 3 different major sponsored works (Aix-Marseille, Nîmes, Lyon 3, you can see everything on git.biblibre.com).
Everything that was in Aix-Marseille branch is now in koha 3.2 . But we have a lot of things done for Lyon 3 (that is live already) and Nimes (that will go live in early july) that are not in koha 3.2

Our main short-term problem is that koha 3.2 is very (very) late. It has been announced one year ago ! We did a lot of testing for Lyon 3 before putting them live. Some of them resulted in some bugfixes that could/should have been submitted to koha mainstream. But, we couldn't. Thus we face a situation where someone fixes in official 3.2 some things we already fixed in our Lyon3 branch (shame on us, but really, we couldn't find time. Delivery to Lyon3 was on time, but is was shorter than short)
The situation is the same with Nimes : they will probably be live (with 3.2 + Lyon3 + Nimes sponsored stuff) before 3.2 is released -going live planned for july 1st, Galen, we challenge you ;-) -.

So, frankly, dealing with our contracts and the official version of Koha has been a pain, and has resulted in a "virtual fork" for Lyon3 (and in 1 month for Nimes). We strongly want to submit our code to the main trunk, but dealing with a so-long-delayed version is really a nightmare (and I understand PTFS deploying "Harley" -even if i share community opinion about the version being advertised when released in a way it can be understood as an official version-). We must find a solution to this situation, or the "virtual fork" will result in a "real fork" that we strongly want to avoid !

As of today, we have something like 400 patches to integrate into koha main trunk !

Technical problem
-----------
With some customers having more than 1 million items (Aix-Marseille : 1247686), and with more and more customers, we reach some limits with the structure of Koha, in term of performances and stability/maintainability. Chris suggested some improvements with koha 3.4, and we have some complementary ideas, I want to share with you, and would like to implement ASAP.

Ideas/proposal
=======
Technical : Koha architecture
---------------------------------
At the begining of may 2010, we staff had a meeting at Marseille to know if it would be possible to make koha easier to maintain and evolve, we are now doing some experiment to split koha into several layers that provides completely independant set of tasks based on the MVC architecture (http://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller).

Koha can be seen as a stack with communication between the closest layers. 4 layers at all (does it make you remember TCP/IP? ;) )

VIEW: interactions with different clients ( browser, REST, XMPP, Z3950, ... )
↓↑
CONTROLLER: Controls the model by transmitting the CRUD tasks requested by the VIEW
↓↑
MODEL: Business objects, as close as possible to the real world
↓↑
PERSISTENCE: Make the Business objects persistent

Plus, we expect to use external tools as often as possible to take experience of other communities and reduce the amount of code we have to debug by ourselves. We think tools exists for every layers, actually we just have to
choose between them:

View:
-------
 HTML::Template, Template Toolkit 2, Template::Declare, SOAP::Lite,
any REST, XMPP, Z3950, SRU/SRW wrapper, ... (Infobot ;))

Controller:
----------
Dancer, Mojo, The Controller parts of existing MVC frameworks (
Jifty, Catalyst, ...), ...

Model:
-------
Any OO Framework ( prefer those which comes with persistence extension as
MooseX::Storage, KiokuDB, ... choose one )

Persistence:
-----------
DBIx::Class, Document DB clients, SQL::Abstract, Jifty::DBI, ....

Ideally, the communication between each layers must be specified, provable by unit testing, and well documented ( pods in the t/* ?).

Today, Koha works in CGI mode, so there is no persistence at all. And having no persistence means we can't implement the MVC model -some tests we did with OO frameworks show very poor performances-, and won't be able to improve performance heavily -the main loss of perf in koha as of today is loading datas that are almost static & should be available immediatly : systempreferences & all the admin tables -.

So: 1st step => remove CGI, to have persistent code, enabling persistent DB connection. We did some tests with Dancer, that are very encouraring, we will send a mail about that soon.
2nd step => Once we will be OK with dancer to get rid of CGI => head to persistence & OO framework. We vote for Moose & KiokuDB, but we still have POC to write to confirm the idea.
3rd step =>view part, by moving to T::T (POC still to do to confirm the idea)

It may look like a koha rewrite but it's not: every parts would be added step by step into koha by wrapping the new design behind the old API during a transition phase. An example of current code

sub GetBranchesInCategory($) {
my ($categorycode) = @_;
my @branches;
my $dbh = C4::Context->dbh();
my $sth = $dbh->prepare(
"SELECT b.branchcode
FROM branchrelations r, branches b
where r.branchcode=b.branchcode and r.categorycode=?"
);
$sth->execute($categorycode);
while ( my $branch = $sth->fetchrow ) {
push @branches, $branch;
}
$sth->finish();
return ( \@branches );
}

would became transition code:

sub GetBranchesInCategory($) {
my $id = shift
or return;

# - Category would be the Business object with a member transition (which can be
# deprecated at the end of transition) that encapsulate every
# persistance specific data used in the C4::* api
# - the find method could be delegated to the DBIx::Class find that returns the
# record associated to the id

# returns
if ( my $category = Category->transition->find($id) ) {
# codes to deal with branches comes de facto with tools as DBIx::Class
# it returns the list of branches for the category
$category->branches;
}
else { undef }
}

You can find in attachement a small graph with all our ideas, and the beginning of a timeline.

ppl reading carefully will see a "changing indexing system (solr)". 4 years after starting with it, we still find zebra being a nightmare, with a lot of problems for unicode (with ICU). We did some tests with solr, and they are very encouraging. We will provide the results soon too. But i'm really enthusiast about what we did in a few hours !

Organizational : Koha team
-------------------------------
    Koha team organisation
    ...............
If we (BibLibre & the Koha community) want to be able to implement such a roadmap (that also includes some new functionnality, as customers sponsors some...), we must have a heavily organised team. I think we must :
* manage release by date (not by features). This is probably the more important part for us (see our "marché public" problem). The timing of the versions would be defined by the companies contributing the most and synched with sponsored devs roadmap. That will be tricky to organize when there are more than one company involved, but, for instance, BibLibre is the largest contributor ( PTFS: you're welcomed to enter the thread ! ). In fact, "a koha 3.4" could be released in a few weeks (look at out master branch, it contains 3.2 + Lyon3 + Nimes sponsored work). A "koha 3.6" could be released in december/february,... a regular release, that's something that is probably "non-negociable" for us.
* define very clear coding guidelines & be very stricts on their use : everything that does not respect the guidelines must be politely but firmly rejected.
* have a very efficient patch workflow : a patch can't be in the wild for weeks. The submitter must have an answer in a very short timeline (is 48H possible ?). ie : We can't rely on only one person to push patches. We must have back-ups for major roles like RM & "patch pusher".
* we could divide Koha in functionnal (serials / acquisition / cataloguing / ...) or technical (perfs / SQL structure / view layer / indexing engine / ...) parts, with some working group for each of them, and something like a monthly report of the WG (with a weekly internal meeting)
* write tests ! technical tests (.t) and functional tests (selenium ?)

    Our involvement (BibLibre) :
    ....................
We want to improve our involvement in Koha, by dedicating Henri-Damien to work at 75% for Koha and be the link between the work done internally by BibLibre, and the work done by everyone else. This mean :
I know we have elected a team recently, and I know chris will do an awesome job as RM (I don't know for others, I didn't saw them in their responsability yet. For chris, i'm the team for 8 years, I know who he is ;-) ), but I think hdl should have a responsability more important than "bug wrangler" to reflect the time he (will) dedicate to the project. Is something like "co-RM" possible ? any other idea ? Any other idea ?

Next step
-----------
We are *really* willing to implement this roadmap and way of doing things. The goal of this mail is to ask everybody involved in Koha development about their opinion: we want our proposal to be a chance for Koha (new deal ?).

On the technical side, stay tuned. Some POC will fall in the next weeks !!!
On the organizational side, the ball is on your side: let's go to update/improve our organization !

-- 
Paul POULAIN
http://www.biblibre.com
Expert en Logiciels Libres pour l'info-doc
Tel : (33) 4 91 81 35 08