[Koha-devel] Search speed

[Paul A]

At 07:16 PM 2/4/2015 -0300, Tomas Cohen Arazi wrote:
>For a fair comparison you should set facetNumRecs = 20 in 
>zebradb/zebra-biblios-dom.xml. Because otherwise use_zebra_facets=0 is 
>only processing the first 20 records, against 1000 from use_zebra_facets=1.

Thanks for the suggestion. All tests stop/start memcached and zebrasrv. 
NYTProf gives:

*** Both set to 20:
Profile of ./opac-search.pl for 21.1s (of 21.3s), executing 1018674 
statements and 252207 subroutine calls in 590 source files and 132 string 
evals.
with 19.3s spent in Net::Z3950::ZOOM::resultset_record (xsub) and the CPU 
(from 'top') is stuck at 100% of a single core for all that time

*** Set 20 (sysprefs) 1000 zebra-biblios-dom.xml:
Profile of ./opac-search.pl for 20.9s (of 21.1s), executing 1018683 
statements and 252793 subroutine calls in 590 source files and 133 string 
evals.
with 19.1s spent in Net::Z3950::ZOOM::resultset_record (xsub) and the CPU 
at 100%

*** Set 20 (sysprefs) 30000 zebra-biblios-dom.xml:
Profile of ./opac-search.pl for 20.9s (of 21.2s), executing 1018669 
statements and 252616 subroutine calls in 590 source files and 134 string 
evals.
with 19.2s spent in Net::Z3950::ZOOM::resultset_record (xsub) and the CPU 
at 100%

But (catastrophe), if I set both for 100 facets (maybe not unreasonable):

Profile of ./opac-search.pl for 67.9s (of 68.2s), executing 1023091 
statements and 255995 subroutine calls in 590 source files and 132 string 
evals, 66.2 secs in resultset_record with CPU 100% for that time.

Conclusion: It's the syspref 'FacetMaxCount' that is the controlling value.

>Regarding the amount of statements/function calls, I don't think they mean 
>too much. Some things got pulled from a +150 line piece of code into 
>several helper functions. I don't think it introduces much overhead.

Maybe not, but several hundred thousand operations, even at a low median of 
240 microsecs, adds up to 3.08 responding in 0.6 - 0.8 secs, and 3.18 
(use_zebra_facets false) needing 1.3 to 1.6

And there are other aspects of "speed" that I'm looking at -- e.g. apache 
2.4 with mod_deflate needs some fine tuning; otherwise images, css, js 
(maybe more) don't necessarily get compressed and add considerably to the 
number of packets, syns, acks, etc and web latency can kick in. But that's 
for another day... ;=)

Again thanks for the suggestion, best -- Paul


>REgards
>Tomas
>
>PS. if you play with facetNumRecs, remember to restart zebra each time you 
>change it.
>
>On Wed, Feb 4, 2015 at 6:22 PM, Paul A <paul.a at navalmarinearchive.com> wrote:
>At 02:02 PM 2/2/2015 +0000, Tomas Cohen Arazi wrote:
>Paul, while you are running your tests, could you please compare this both 
>scenarios:
>- <use_zebra_facets>1</use_zebra_facets>
>- <use_zebra_facets>0</use_zebra_facets>
>Thanks
>PS: I have found that searchResults is too slow too.
>
>
>I have posted notes, results and some NYTProf examples at 
><http://navalmarinearchive.com/z_koha/> -- the results are repeatable (I 
>must have run NYTProf nearly 200 times.)
>
>Provisional conclusions are that <use_zebra_facets> = 1 (true) is not 
>usable in production for a search that finds > 20k results. For smaller 
>results, it might be acceptable, but is still appreciably slower than 
>3.08. Under the best circumstances (0-5 results) 3.08 responds in < 0.7 
>secs, 3.18 in 1.3 secs.  At the other extreme (25k results), 3.08 
>requires 1.07 secs, 3.18 21+ secs.
>
>I have defined an "optimal" 3.18 combination (see green line, Graph 2) 
>which runs at about half the speed of 3.08, but does not give 'Series', 
>'Places' and 'Titles' in the facets.
>
>One point that jumped out is that a best case 3.08 search executes 431207 
>statements and 76556 subroutine calls in 297 source files and 81 string 
>evals whereas a worst case 3.18 executes 1238167 statements and 309905 
>subroutine calls in 591 source files and 132 string evals. (Even the worst 
>case 308 appears much more efficient than the best case 3.18.)
>
>The use of -T to start Zebra is non-conclusive -- it vaguely slows down 
>smaller result sets and vaguely speeds up larger result sets (testing is 
>repeatable.)Â  Robin was of course quite correct in his recent email, but 
>I cannot explain even the minor variations experienced.
>
>I'm not at all certain how to file this as a bug, so will await any comments.
>
>Best -- Paul