[Koha-devel] BibLibre strategy for Koha 3.4 and next versions

[Paul Poulain]

Hi,

In this mail, I want to share with you our situation, our problems, and
our ideas.
You'll be probably surprised by this mail. Please consider it's a good
news. It's not a threat - not my cup of tea (frenchism suspected) - .
But I think things must be done this way, so I share it with you, and
I'm impatiently waiting for koha-community reaction & thoughts.

BibLibre situation
===========
We are very successful. Koha is really a success in France. One french
university is now live, we are on the way to migrate a very large public
library (500k items), 3 other universities should go live in September,
and a lot of business coming.
We are 11, are on our way to hire 3 new ppl, have on board Philippe
Chabanon, that is very experienced in managing a company, is fully
enthusiastic about free softwares and community, and knows a lot of
people in the french library world.
Is everything perfect ? nope.

Our problems
=========
Organizational problem
------------
Some of our clients sponsor new developments. Those are done under
"marché public" rules, that usually define a very strict date (if we
fail, then we have to pay fines, based on how many days late we are !).
That's why we started 3 different branches for our 3 different major
sponsored works (Aix-Marseille, Nîmes, Lyon 3, you can see everything on
git.biblibre.com).
Everything that was in Aix-Marseille branch is now in koha 3.2 . But we
have a lot of things done for Lyon 3 (that is live already) and Nimes
(that will go live in early july) that are not in koha 3.2

Our main short-term problem is that koha 3.2 is very (very) late. It has
been announced one year ago ! We did a lot of testing for Lyon 3 before
putting them live. Some of them resulted in some bugfixes that
could/should have been submitted to koha mainstream. But, we couldn't.
Thus we face a situation where someone fixes in official 3.2 some things
we already fixed in our Lyon3 branch (shame on us, but really, we
couldn't find time. Delivery to Lyon3 was on time, but is was shorter
than short)
The situation is the same with Nimes : they will probably be live (with
3.2 + Lyon3 + Nimes sponsored stuff) before 3.2 is released -going live
planned for july 1st, Galen, we challenge you ;-) -.

So, frankly, dealing with our contracts and the official version of Koha
has been a pain, and has resulted in a "virtual fork" for Lyon3 (and in
1 month for Nimes). We strongly want to submit our code to the main
trunk, but dealing with a so-long-delayed version is really a nightmare
(and I understand PTFS deploying "Harley" -even if i share community
opinion about the version being advertised when released in a way it can
be understood as an official version-). We must find a solution to this
situation, or the "virtual fork" will result in a "real fork" that we
strongly want to avoid !

As of today, we have something like 400 patches to integrate into koha
main trunk !

Technical problem
-----------
With some customers having more than 1 million items (Aix-Marseille :
1247686), and with more and more customers, we reach some limits with
the structure of Koha, in term of performances and
stability/maintainability. Chris suggested some improvements with koha
3.4, and we have some complementary ideas, I want to share with you, and
would like to implement ASAP.

Ideas/proposal
=======
Technical : Koha architecture
---------------------------------
At the begining of may 2010, we staff had a meeting at Marseille to know
if it would be possible to make koha easier to maintain and evolve, we
are now doing some experiment to split koha into several layers that
provides completely independant set of tasks based on the MVC
architecture
(http://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller).

Koha can be seen as a stack with communication between the closest
layers. 4 layers at all (does it make you remember TCP/IP? ;) )

VIEW: interactions with different clients ( browser, REST, XMPP, Z3950,
... )
??
CONTROLLER: Controls the model by transmitting the CRUD tasks requested
by the VIEW
??
MODEL: Business objects, as close as possible to the real world
??
PERSISTENCE: Make the Business objects persistent

Plus, we expect to use external tools as often as possible to take
experience of other communities and reduce the amount of code we have to
debug by ourselves. We think tools exists for every layers, actually we
just have to
choose between them:

View:
-------
HTML::Template, Template Toolkit 2, Template::Declare, SOAP::Lite,
any REST, XMPP, Z3950, SRU/SRW wrapper, ... (Infobot ;))

Controller:
----------
Dancer, Mojo, The Controller parts of existing MVC frameworks (
Jifty, Catalyst, ...), ...

Model:
-------
Any OO Framework ( prefer those which comes with persistence extension as
MooseX::Storage, KiokuDB, ... choose one )

Persistence:
-----------
DBIx::Class, Document DB clients, SQL::Abstract, Jifty::DBI, ....

Ideally, the communication between each layers must be specified,
provable by unit testing, and well documented ( pods in the t/* ?).

Today, Koha works in CGI mode, so there is no persistence at all. And
having no persistence means we can't implement the MVC model -some tests
we did with OO frameworks show very poor performances-, and won't be
able to improve performance heavily -the main loss of perf in koha as of
today is loading datas that are almost static & should be available
immediatly : systempreferences & all the admin tables -.

So: 1st step => remove CGI, to have persistent code, enabling persistent
DB connection. We did some tests with Dancer, that are very encouraring,
we will send a mail about that soon.
2nd step => Once we will be OK with dancer to get rid of CGI => head to
persistence & OO framework. We vote for Moose & KiokuDB, but we still
have POC to write to confirm the idea.
3rd step =>view part, by moving to T::T (POC still to do to confirm the
idea)

It may look like a koha rewrite but it's not: every parts would be added
step by step into koha by wrapping the new design behind the old API
during a transition phase. An example of current code

sub GetBranchesInCategory($) {
my ($categorycode) = @_;
my @branches;
my $dbh = C4::Context->dbh();
my $sth = $dbh->prepare(
"SELECT b.branchcode
FROM branchrelations r, branches b
where r.branchcode=b.branchcode and r.categorycode=?"
);
$sth->execute($categorycode);
while ( my $branch = $sth->fetchrow ) {
push @branches, $branch;
}
$sth->finish();
return ( \@branches );
}

would became transition code:

sub GetBranchesInCategory($) {
my $id = shift
or return;

# - Category would be the Business object with a member transition
(which can be
# deprecated at the end of transition) that encapsulate every
# persistance specific data used in the C4::* api
# - the find method could be delegated to the DBIx::Class find that
returns the
# record associated to the id

# returns
if ( my $category = Category->transition->find($id) ) {
# codes to deal with branches comes de facto with tools as DBIx::Class
# it returns the list of branches for the category
$category->branches;
}
else { undef }
}

You can find in attachement a small graph with all our ideas, and the
beginning of a timeline.

ppl reading carefully will see a "changing indexing system (solr)". 4
years after starting with it, we still find zebra being a nightmare,
with a lot of problems for unicode (with ICU). We did some tests with
solr, and they are very encouraging. We will provide the results soon
too. But i'm really enthusiast about what we did in a few hours !

Organizational : Koha team
-------------------------------
    Koha team organisation
    ...............
If we (BibLibre & the Koha community) want to be able to implement such
a roadmap (that also includes some new functionnality, as customers
sponsors some...), we must have a heavily organised team. I think we must :
* manage release by date (not by features). This is probably the more
important part for us (see our "marché public" problem). The timing of
the versions would be defined by the companies contributing the most and
synched with sponsored devs roadmap. That will be tricky to organize
when there are more than one company involved, but, for instance,
BibLibre is the largest contributor ( PTFS: you're welcomed to enter the
thread ! ). In fact, "a koha 3.4" could be released in a few weeks (look
at out master branch, it contains 3.2 + Lyon3 + Nimes sponsored work). A
"koha 3.6" could be released in december/february,... a regular release,
that's something that is probably "non-negociable" for us.
* define very clear coding guidelines & be very stricts on their use :
everything that does not respect the guidelines must be politely but
firmly rejected.
* have a very efficient patch workflow : a patch can't be in the wild
for weeks. The submitter must have an answer in a very short timeline
(is 48H possible ?). ie : We can't rely on only one person to push
patches. We must have back-ups for major roles like RM & "patch pusher".
* we could divide Koha in functionnal (serials / acquisition /
cataloguing / ...) or technical (perfs / SQL structure / view layer /
indexing engine / ...) parts, with some working group for each of them,
and something like a monthly report of the WG (with a weekly internal
meeting)
* write tests ! technical tests (.t) and functional tests (selenium ?)

    Our involvement (BibLibre) :
    ....................
We want to improve our involvement in Koha, by dedicating Henri-Damien
to work at 75% for Koha and be the link between the work done internally
by BibLibre, and the work done by everyone else. This mean :

    * publish our RFPs as soon as possible
    * gather all the work done every month and communicate about that on
      the wiki and blog and translate on koha-fr.org
    * manage patches coming out from BibLibre, and coming out from
      everyone else (to deal as soon as possible with any problem)

I know we have elected a team recently, and I know chris will do an
awesome job as RM (I don't know for others, I didn't saw them in their
responsability yet. For chris, i'm the team for 8 years, I know who he
is ;-) ), but I think hdl should have a responsability more important
than "bug wrangler" to reflect the time he (will) dedicate to the
project. Is something like "co-RM" possible ? any other idea ? Any other
idea ?

Next step
-----------
We are *really* willing to implement this roadmap and way of doing
things. The goal of this mail is to ask everybody involved in Koha
development about their opinion: we want our proposal to be a chance for
Koha (new deal ?).

On the technical side, stay tuned. Some POC will fall in the next weeks !!!
On the organizational side, the ball is on your side: let's go to
update/improve our organization !

-- 
Paul POULAIN
http://www.biblibre.com
Expert en Logiciels Libres pour l'info-doc
Tel : (33) 4 91 81 35 08

-------------- section suivante --------------
Une pi?ce jointe HTML a ?t? nettoy?e...
URL: </pipermail/koha-devel/attachments/20100602/1a238794/attachment-0001.htm>