[Koha-devel] Restructuring C4

[Clay Fouts]

Hello, Ian.

Disentangling these dependency cycles is essential for Koha's development to
continue on in a meaningful way. As is, changing one small section of code
often requires changing a very similar chunk of code in five other areas (if
you know about all five of those other areas, which you probably don't,
which means you've introduced action-at-a-distance bugs). The approach of
separating the DB schema layer out from a model layer is a solid first step.
Plus, the deep entangling makes reliable testing extremely difficult,
meaning that the AAD discrepancies are difficult to detect in an automated
fashion. Separating out the view and controller pieces is also essential for
testing.

However, this step by itself is only going to marginally optimize for
execution time. It will help inasmuch as it may reduce the number of modules
that get loaded for any given operation, and that's a boon. But each module
only ever gets loaded and its BEGIN block processed once per script
execution cycle, regardless of how many times it is "used." At the same
time, many of the efficiency issues are the product of processing the same
data a thousand times instead of once, because each model-layer subroutine
makes direct calls to the database instead of normalizing those accesses
into a schema management layer that can uniformly cache (and un-cache!)
data.

Ultimately, migrating from CGI to persistence is the big step that needs to
be taken for improving speed, but that step cannot be taken until the API is
disentangled, making execution more predictable.

Regards,
Clay


2011/3/30 Ian Walls <ian.walls at bywatersolutions.com>

> Fellow Developers,
>
>
> Last night, I stayed up late running circ/circulation.pl through NYTProf,
> to get an idea where we may be able to optimize circulation for speed.
> After much frustration (darn session IDs...), I was able to get a report.
> The results were... not exactly what I expected.
>
> It seems that much of the time spent running circ/circulation.pl is spent
> BEGINing the various C4 modules, multiple times.  C4::Items, for example, is
> BEGuN 14 times in the execution of circ/circulation.pl.  More stats:
>
> C4::Accounts - 8 BEGINs
> C4::Acquisition - 14 BEGINs
> C4::Auth        - 14 BEGINs
> C4::Biblio       - 14 BEGINs
> C4::Branch     - 5 BEGINs
> C4::Circulation - 22 BEGINs
> C4::Context     - 18 BEGINs
> C4::Dates       - 12 BEGINs
>
> ... and so on.  Even modules like C4::XSLT, C4::SMS, C4::Suggestions, and
> C4::Search::PazPar2 are BEGuN several times, without any calls to any of
> their subroutines.  Some modules, it seems, don't have an END at the end...
>
> I was also able to get a Graphviz file out, showing all the package calls
> (folks may remember my keen interest in such things from KohaCon).  Running
> it through sccmap to detect cycles, I found a very long one between several
> of the modules.  About 8 packages long.
>
> All this indicate to me that we REALLY need to start thinking about
> imposing some kind of structure on C4.  I'm thinking something with
> two-levels:
>
> Level 1 :  Calls to the database, other direct interaction with stored data
> Level 2:   Calls to Level 1 functions to manipulate data
>
> all our scripts would then be modified to call only Level 2, which would
> take care of talking to Level 1.  At the worst, a particular Level 1 package
> would be BEGuN as many times as there were Level 2 packages BEGuN in the
> script (and we could certainly do better).  It would eliminate all the
> cycles, as well.
>
> I can provide access to my report for anyone who needs it, or more details
> about how I obtained it.  Whatever any of you need to keep this conversation
> going.  This is a big deal, in my opinion, and we need to start detangling
> our code before too much long, or we're going to get seriously bogged down.
>
> Cheers,
>
>
> -Ian
>
> --
> Ian Walls
> Lead Development Specialist
> ByWater Solutions
> Phone # (888) 900-8944
> http://bywatersolutions.com
> ian.walls at bywatersolutions.com
> Twitter: @sekjal
>
> _______________________________________________
> Koha-devel mailing list
> Koha-devel at lists.koha-community.org
> http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
> website : http://www.koha-community.org/
> git : http://git.koha-community.org/
> bugs : http://bugs.koha-community.org/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/koha-devel/attachments/20110330/17673dc8/attachment-0001.htm>