On Wed, Mar 30, 2011 at 1:17 PM, Clay Fouts <cfouts@liblime.com> wrote:

Hello, Ian.

Disentangling these dependency cycles is essential for Koha's development to continue on in a meaningful way. As is, changing one small section of code often requires changing a very similar chunk of code in five other areas (if you know about all five of those other areas, which you probably don't, which means you've introduced action-at-a-distance bugs). The approach of separating the DB schema layer out from a model layer is a solid first step. Plus, the deep entangling makes reliable testing extremely difficult, meaning that the AAD discrepancies are difficult to detect in an automated fashion. Separating out the view and controller pieces is also essential for testing.

However, this step by itself is only going to marginally optimize for execution time. It will help inasmuch as it may reduce the number of modules that get loaded for any given operation, and that's a boon. But each module only ever gets loaded and its BEGIN block processed once per script execution cycle, regardless of how many times it is "used." At the same time, many of the efficiency issues are the product of processing the same data a thousand times instead of once, because each model-layer subroutine makes direct calls to the database instead of normalizing those accesses into a schema management layer that can uniformly cache (and un-cache!) data.

Ultimately, migrating from CGI to persistence is the big step that needs to be taken for improving speed, but that step cannot be taken until the API is disentangled, making execution more predictable.

Regards,
Clay

2011/3/30 Ian Walls <ian.walls@bywatersolutions.com>

Fellow Developers,

Last night, I stayed up late running circ/circulation.pl through NYTProf, to get an idea where we may be able to optimize circulation for speed. After much frustration (darn session IDs...), I was able to get a report. The results were... not exactly what I expected.

It seems that much of the time spent running circ/circulation.pl is spent BEGINing the various C4 modules, multiple times. C4::Items, for example, is BEGuN 14 times in the execution of circ/circulation.pl. More stats:

C4::Accounts - 8 BEGINs
C4::Acquisition - 14 BEGINs
C4::Auth        - 14 BEGINs
C4::Biblio       - 14 BEGINs
C4::Branch     - 5 BEGINs
C4::Circulation - 22 BEGINs
C4::Context     - 18 BEGINs
C4::Dates       - 12 BEGINs

... and so on. Even modules like C4::XSLT, C4::SMS, C4::Suggestions, and C4::Search::PazPar2 are BEGuN several times, without any calls to any of their subroutines. Some modules, it seems, don't have an END at the end...

I was also able to get a Graphviz file out, showing all the package calls (folks may remember my keen interest in such things from KohaCon). Running it through sccmap to detect cycles, I found a very long one between several of the modules. About 8 packages long.

All this indicate to me that we REALLY need to start thinking about imposing some kind of structure on C4. I'm thinking something with two-levels:

Level 1 : Calls to the database, other direct interaction with stored data
Level 2:   Calls to Level 1 functions to manipulate data

all our scripts would then be modified to call only Level 2, which would take care of talking to Level 1. At the worst, a particular Level 1 package would be BEGuN as many times as there were Level 2 packages BEGuN in the script (and we could certainly do better). It would eliminate all the cycles, as well.

I can provide access to my report for anyone who needs it, or more details about how I obtained it. Whatever any of you need to keep this conversation going. This is a big deal, in my opinion, and we need to start detangling our code before too much long, or we're going to get seriously bogged down.

Cheers,

-Ian

--
Ian Walls
Lead Development Specialist
ByWater Solutions
Phone # (888) 900-8944
http://bywatersolutions.com
ian.walls@bywatersolutions.com
Twitter: @sekjal

_______________________________________________
Koha-devel mailing list
Koha-devel@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/