Re: [Koha-devel] [Koha-patches] [PATCH] Disable _remove_stopwords in Search.pm to improve performances
On Wed, Jul 2, 2008 at 7:28 AM, Frederic Demians <f.demians@tamil.fr> wrote:
_remove_stopwords takes 28% of all the time spent to do search. This patch disable this function. Stop words should be handle directly by Zebra, shouldn't it?
Hi Frédéric - I'm all in favor of the current work to profile our application and attempt to improve it. This does look like a place where we can improve our efficiency. Have you tested to see if the zebra engine does handle the stopwords correctly? I didn't know that it did that. What should happen in the NoZebra case? Also, if we're going to stop using this sub, should we remove it and all calls to it instead of just shunting it? Thanks! -Andy _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
Andrew Moore a écrit :
I'm all in favor of the current work to profile our application and attempt to improve it. This does look like a place where we can improve our efficiency.
Have you tested to see if the zebra engine does handle the stopwords correctly? I didn't know that it did that.
What should happen in the NoZebra case?
it will break NoZebra.
Also, if we're going to stop using this sub, should we remove it and all calls to it instead of just shunting it?
I definetly think we should NOT shunting this sub. zebra can deal with stopwords, but it's in a file that the librarian can't edit easily. With the stopword, it's easy & can be done at any time Note : the C4::Context has been written years ago, and was supposed to deal with subsequent calls to his objects. was it wrong for years, or did someone change something in the wrong way recently ? It had been written by a monger, so never has any doubt it was efficient... -- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc Tel : 04 91 31 45 19 _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
Have you tested to see if the zebra engine does handle the stopwords correctly? I didn't know that it did that.
No I haven't tested and I don't know also if Zebra supports stop words. I can't find any information on the subject. Stop words have less and less interest with zebra mechanisms like relevance ranking or field weighting. In Zebra ccl.properties, there is a st-word-list line?... And google gives that: http://lists.indexdata.dk/pipermail/zebralist/2006-February/001001.html
What should happen in the NoZebra case?
Good question.
Also, if we're going to stop using this sub, should we remove it and all calls to it instead of just shunting it?
Yes, stop stopwords :-) Here is a summary of my tests: opac-main.pl page: * without mod_perl: 1.12s to load * with mod_perl: 0.3s opac-search.pl, with mod_perl, a search hitting 300 records in a 200,000 biblio records DB: * at the beginning, without any optimization: >4s * with my patches: 1.8s. Patch on: o C4::Context reading Koha config file multiple time o stop words o biblio framework caching application-wide * it seems that optimization can be done on authorized values -- Frédéric _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
Hi, On Wed, Jul 2, 2008 at 7:32 AM, Frederic Demians <frederic@tamil.fr> wrote:
Have you tested to see if the zebra engine does handle the stopwords correctly? I didn't know that it did that.
No I haven't tested and I don't know also if Zebra supports stop words.
I found the first part of your sentence troubling, but perhaps I worry too much. :) A good case can be made that stopwords are (or could be) irrelevant when Zebra is used, but surely they are needed for NoZebra to perform reasonably well. At this point, patches to improve performance should not change functionality, and ideally a test case should accompany each major performance fix to verify that functional behavior has not in fact changed. Perhaps you could take another look at _remove_stopwords() to see if a faster implementation can be written. Regards, Galen -- Galen Charlton Koha Application Developer LibLime galen.charlton@liblime.com p: 1-888-564-2457 x709 _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
Have you tested to see if the zebra engine does handle the stopwords correctly? I didn't know that it did that. No I haven't tested and I don't know also if Zebra supports stop words.
I found the first part of your sentence troubling, but perhaps I worry too much. :)
I haven't tested Zebra stopwords support for good reason: it seems not to exist. I have tested my patch :) Does it alleviate ambiguity?
A good case can be made that stopwords are (or could be) irrelevant when Zebra is used, but surely they are needed for NoZebra to perform reasonably well.
I agree. So something like: return unless NoZebra...
At this point, patches to improve performance should not change functionality, and ideally a test case should accompany each major performance fix to verify that functional behavior has not in fact changed. Perhaps you could take another look at _remove_stopwords() to see if a faster implementation can be written.
The point of this patch is to underline and prove that _remove_stopwords function take 1/3 of total response time for a search request. At the beginning of my quest (sic), there is the observation that opac search response time was not so good (4 s). I was able to do some performance testing, but I'm not necessarily the more qualified to touch a function like this one. Will give a try whatever. Thanks. -- Frédéric _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha.org http://lists.koha.org/mailman/listinfo/koha-devel
participants (4)
-
Andrew Moore -
Frederic Demians -
Galen Charlton -
Paul POULAIN