I have progressed a little further on this subject using multiple parallel tests for from 0 to ~25k biblios on a new, relatively fast server. In brief: 3.08 averages 0.88 secs per search 3.18 averages 5.06 secs for the same searches, but... 3.18 averages only 1.52 secs if I turn off facets. Facets appear to have a huge overhead in various components of Zebra, to the point where a single core of a multiple processor CPU gets overwhelmed, but does not share the load with other totally idle cores. Intel tell me that this is not fixable through any hardware/BIOS config and they are not aware of a Linux "standard library" that deals with processor affinity. I've searched around a little and found mentions for c and FORTRAN (marvelous old memories for me!) but so far nothing totally specific for Perl. taskset and sched_setaffinity might be usable at system level, and I might have a look at pthread_setaffinity_np.3.gz I've posted test results, notes and three sets of NYTProf analysis at <http://navalmarinearchive.com/z_koha/> and would appreciate if anyone has any suggestions or comparative tests and outcomes. Thanks and best regards -- Paul
Paul A schreef op do 29-01-2015 om 19:39 [-0500]:
Facets appear to have a huge overhead in various components of Zebra, to the point where a single core of a multiple processor CPU gets overwhelmed, but does not share the load with other totally idle cores.
That's not how multiprocessor systems work. They can only spread load amongst other cores if the software is multithreaded/multiprocess, which I suspect zebra isn't, at least within a single request. If it does all its work in a single thread, then it can't spread across multiple cores.
Intel tell me that this is not fixable through any hardware/BIOS config and they are not aware of a Linux "standard library" that deals with processor affinity. I've searched around a little and found mentions for c and FORTRAN (marvelous old memories for me!) but so far nothing totally specific for Perl.
Zebra is not Perl, zebra is (I think) C. But still, unless it's written with multithreading in mind, you can't do anything about it.
taskset and sched_setaffinity might be usable at system level
Unless you have very specific use cases (which you don't), these won't help.
, and I might have a look at pthread_setaffinity_np.3.gz
That will only help if you plan on rewriting zebra to use pthreads. I don't recommend rewriting zebra. -- Robin Sheat Catalyst IT Ltd. ✆ +64 4 803 2204 GPG: 5FA7 4B49 1E4D CAA4 4C38 8505 77F5 B724 F871 3BDF
You can launch Zebra multi-threaded using : zebrasrv -T Le 30/01/2015 02:59, Robin Sheat a écrit :
Paul A schreef op do 29-01-2015 om 19:39 [-0500]:
Facets appear to have a huge overhead in various components of Zebra, to the point where a single core of a multiple processor CPU gets overwhelmed, but does not share the load with other totally idle cores.
That's not how multiprocessor systems work.
They can only spread load amongst other cores if the software is multithreaded/multiprocess, which I suspect zebra isn't, at least within a single request. If it does all its work in a single thread, then it can't spread across multiple cores.
Intel tell me that this is not fixable through any hardware/BIOS config and they are not aware of a Linux "standard library" that deals with processor affinity. I've searched around a little and found mentions for c and FORTRAN (marvelous old memories for me!) but so far nothing totally specific for Perl.
Zebra is not Perl, zebra is (I think) C. But still, unless it's written with multithreading in mind, you can't do anything about it.
taskset and sched_setaffinity might be usable at system level
Unless you have very specific use cases (which you don't), these won't help.
, and I might have a look at pthread_setaffinity_np.3.gz
That will only help if you plan on rewriting zebra to use pthreads.
I don't recommend rewriting zebra.
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Fridolin SOMERS Biblibre - Pôles support et système fridolin.somers@biblibre.com
At 05:40 PM 1/30/2015 +0100, Fridolin SOMERS wrote:
You can launch Zebra multi-threaded using : zebrasrv -T
Fridolin - merci mille fois -- first impressions are most positive. I'll try and find time later today to quantify the results and put them in context of my previous tests. [vi /usr/share/koha/bin/koha-zebra-ctl.sh line 40 to ZEBRAOPTIONS="-T -v none,fatal,warn"] J'aurais bien aimer trouver ça tout seul ... mais encore une fois avec mes remerciments. Meilleures amitiés et a+ Paul
Le 30/01/2015 02:59, Robin Sheat a écrit :
Paul A schreef op do 29-01-2015 om 19:39 [-0500]:
Facets appear to have a huge overhead in various components of Zebra, to the point where a single core of a multiple processor CPU gets overwhelmed, but does not share the load with other totally idle cores.
That's not how multiprocessor systems work.
They can only spread load amongst other cores if the software is multithreaded/multiprocess, which I suspect zebra isn't, at least within a single request. If it does all its work in a single thread, then it can't spread across multiple cores.
Intel tell me that this is not fixable through any hardware/BIOS config and they are not aware of a Linux "standard library" that deals with processor affinity. I've searched around a little and found mentions for c and FORTRAN (marvelous old memories for me!) but so far nothing totally specific for Perl.
Zebra is not Perl, zebra is (I think) C. But still, unless it's written with multithreading in mind, you can't do anything about it.
taskset and sched_setaffinity might be usable at system level
Unless you have very specific use cases (which you don't), these won't help.
, and I might have a look at pthread_setaffinity_np.3.gz
That will only help if you plan on rewriting zebra to use pthreads.
I don't recommend rewriting zebra.
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Fridolin SOMERS Biblibre - Pôles support et système fridolin.somers@biblibre.com _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
--- Maritime heritage and history, preservation and conservation, research and education through the written word and the arts. <http://NavalMarineArchive.com> and <http://UltraMarine.ca>
Greetings, If there is no negative effect, and only positive effects, I'd be more than willing to submit a patch to default with a -T. :) GPML, Mark Tompsett -----Original Message----- From: Paul A Sent: Friday, January 30, 2015 3:19 PM To: Fridolin SOMERS ; koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Search speed At 05:40 PM 1/30/2015 +0100, Fridolin SOMERS wrote:
You can launch Zebra multi-threaded using : zebrasrv -T
Fridolin - merci mille fois -- first impressions are most positive. I'll try and find time later today to quantify the results and put them in context of my previous tests. [vi /usr/share/koha/bin/koha-zebra-ctl.sh line 40 to ZEBRAOPTIONS="-T -v none,fatal,warn"] J'aurais bien aimer trouver ça tout seul ... mais encore une fois avec mes remerciments. Meilleures amitiés et a+ Paul
Le 30/01/2015 02:59, Robin Sheat a écrit :
Paul A schreef op do 29-01-2015 om 19:39 [-0500]:
Facets appear to have a huge overhead in various components of Zebra, to the point where a single core of a multiple processor CPU gets overwhelmed, but does not share the load with other totally idle cores.
That's not how multiprocessor systems work.
They can only spread load amongst other cores if the software is multithreaded/multiprocess, which I suspect zebra isn't, at least within a single request. If it does all its work in a single thread, then it can't spread across multiple cores.
Intel tell me that this is not fixable through any hardware/BIOS config and they are not aware of a Linux "standard library" that deals with processor affinity. I've searched around a little and found mentions for c and FORTRAN (marvelous old memories for me!) but so far nothing totally specific for Perl.
Zebra is not Perl, zebra is (I think) C. But still, unless it's written with multithreading in mind, you can't do anything about it.
taskset and sched_setaffinity might be usable at system level
Unless you have very specific use cases (which you don't), these won't help.
, and I might have a look at pthread_setaffinity_np.3.gz
That will only help if you plan on rewriting zebra to use pthreads.
I don't recommend rewriting zebra.
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Fridolin SOMERS Biblibre - Pôles support et système fridolin.somers@biblibre.com _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
--- Maritime heritage and history, preservation and conservation, research and education through the written word and the arts. <http://NavalMarineArchive.com> and <http://UltraMarine.ca> _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
At 04:53 PM 1/30/2015 -0500, Mark Tompsett wrote:
Greetings,
If there is no negative effect, and only positive effects, I'd be more than willing to submit a patch to default with a -T. :)
I'm looking for "negatives" right now. Give me another little while, and I'll do a bug/enhancement myself with all the numbers (a bit of a struggle, NYTProf doesn't appear to like multi-threading.) Best -- Paul
GPML, Mark Tompsett
-----Original Message----- From: Paul A Sent: Friday, January 30, 2015 3:19 PM To: Fridolin SOMERS ; koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Search speed
At 05:40 PM 1/30/2015 +0100, Fridolin SOMERS wrote:
You can launch Zebra multi-threaded using : zebrasrv -T
Fridolin - merci mille fois -- first impressions are most positive. I'll try and find time later today to quantify the results and put them in context of my previous tests.
[vi /usr/share/koha/bin/koha-zebra-ctl.sh line 40 to ZEBRAOPTIONS="-T -v none,fatal,warn"]
J'aurais bien aimer trouver ça tout seul ... mais encore une fois avec mes remerciments.
Meilleures amitiés et a+ Paul
Le 30/01/2015 02:59, Robin Sheat a écrit :
Paul A schreef op do 29-01-2015 om 19:39 [-0500]:
Facets appear to have a huge overhead in various components of Zebra, to the point where a single core of a multiple processor CPU gets overwhelmed, but does not share the load with other totally idle cores.
That's not how multiprocessor systems work.
They can only spread load amongst other cores if the software is multithreaded/multiprocess, which I suspect zebra isn't, at least within a single request. If it does all its work in a single thread, then it can't spread across multiple cores.
Intel tell me that this is not fixable through any hardware/BIOS config and they are not aware of a Linux "standard library" that deals with processor affinity. I've searched around a little and found mentions for c and FORTRAN (marvelous old memories for me!) but so far nothing totally specific for Perl.
Zebra is not Perl, zebra is (I think) C. But still, unless it's written with multithreading in mind, you can't do anything about it.
taskset and sched_setaffinity might be usable at system level
Unless you have very specific use cases (which you don't), these won't help.
, and I might have a look at pthread_setaffinity_np.3.gz
That will only help if you plan on rewriting zebra to use pthreads.
I don't recommend rewriting zebra.
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Fridolin SOMERS Biblibre - Pôles support et système fridolin.somers@biblibre.com _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
--- Maritime heritage and history, preservation and conservation, research and education through the written word and the arts. <http://NavalMarineArchive.com> and <http://UltraMarine.ca>
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
--- Maritime heritage and history, preservation and conservation, research and education through the written word and the arts. <http://NavalMarineArchive.com> and <http://UltraMarine.ca>
Paul, while you are running your tests, could you please compare this both scenarios: - <use_zebra_facets>1</use_zebra_facets> - <use_zebra_facets>0</use_zebra_facets> Thanks PS: I have found that searchResults is too slow too. El Fri Jan 30 2015 at 19:34:41, Paul A (<paul.a@navalmarinearchive.com>) escribió:
At 04:53 PM 1/30/2015 -0500, Mark Tompsett wrote:
Greetings,
If there is no negative effect, and only positive effects, I'd be more than willing to submit a patch to default with a -T. :)
I'm looking for "negatives" right now. Give me another little while, and I'll do a bug/enhancement myself with all the numbers (a bit of a struggle, NYTProf doesn't appear to like multi-threading.)
Best -- Paul
GPML, Mark Tompsett
-----Original Message----- From: Paul A Sent: Friday, January 30, 2015 3:19 PM To: Fridolin SOMERS ; koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Search speed
At 05:40 PM 1/30/2015 +0100, Fridolin SOMERS wrote:
You can launch Zebra multi-threaded using : zebrasrv -T
Fridolin - merci mille fois -- first impressions are most positive. I'll try and find time later today to quantify the results and put them in context of my previous tests.
[vi /usr/share/koha/bin/koha-zebra-ctl.sh line 40 to ZEBRAOPTIONS="-T -v none,fatal,warn"]
J'aurais bien aimer trouver ça tout seul ... mais encore une fois avec mes remerciments.
Meilleures amitiés et a+ Paul
Le 30/01/2015 02:59, Robin Sheat a écrit :
Paul A schreef op do 29-01-2015 om 19:39 [-0500]:
Facets appear to have a huge overhead in various components of Zebra, to the point where a single core of a multiple processor CPU gets overwhelmed, but does not share the load with other totally idle cores.
That's not how multiprocessor systems work.
They can only spread load amongst other cores if the software is multithreaded/multiprocess, which I suspect zebra isn't, at least within a single request. If it does all its work in a single thread, then it can't spread across multiple cores.
Intel tell me that this is not fixable through any hardware/BIOS config and they are not aware of a Linux "standard library" that deals with processor affinity. I've searched around a little and found mentions for c and FORTRAN (marvelous old memories for me!) but so far nothing totally specific for Perl.
Zebra is not Perl, zebra is (I think) C. But still, unless it's written with multithreading in mind, you can't do anything about it.
taskset and sched_setaffinity might be usable at system level
Unless you have very specific use cases (which you don't), these won't help.
, and I might have a look at pthread_setaffinity_np.3.gz
That will only help if you plan on rewriting zebra to use pthreads.
I don't recommend rewriting zebra.
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Fridolin SOMERS Biblibre - Pôles support et système fridolin.somers@biblibre.com _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
--- Maritime heritage and history, preservation and conservation, research and education through the written word and the arts. <http://NavalMarineArchive.com> and <http://UltraMarine.ca>
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
--- Maritime heritage and history, preservation and conservation, research and education through the written word and the arts. <http://NavalMarineArchive.com> and <http://UltraMarine.ca>
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
Fridolin SOMERS schreef op vr 30-01-2015 om 17:40 [+0100]:
You can launch Zebra multi-threaded using : zebrasrv -T
From the documentation: -T Operate the server in threaded mode. The server creates a thread for each connection rather than a fork a process. Only available on UNIX systems that offers POSIX threads. So, this will have no real gain as it doesn't run a single request multithreaded. Process creation in Linux is almost free (this isn't the case on some other operating systems), so it shouldn't have any significant effect. The benefit of multiprocess rather than multithreaded is that if one process crashes, it doesn't take all the other ones with it. Mark Tompsett schreef op vr 30-01-2015 om 16:53 [-0500]:
If there is no negative effect, and only positive effects, I'd be more than willing to submit a patch to default with a -T. :)
I would encourage reading the documentation first, so that you understand what it actually does. Paul A schreef op vr 30-01-2015 om 17:34 [-0500]:
I'm looking for "negatives" right now. Give me another little while, and I'll do a bug/enhancement myself with all the numbers (a bit of a struggle, NYTProf doesn't appear to like multi-threading.)
You're still misunderstanding how everything ties together, and this means you won't understand the results you get. * NYTProf doesn't have to care about multithreading because it is profiling Koha and Koha is not multithreaded. * NYTProf doesn't profile zebra because zebra is not written in Perl and is not part of Koha. * Zebra is a service that Koha asks a question to, and gets a result. It could be (and sometimes is) on a totally different server. NYTProf doesn't know and doesn't care whether zebrasrv forks itself or spawns a thread when it gets a connection, neither does Koha. The option to make zebrasrv use threads is purely a system administration level decision made with awareness of what it means. -- Robin Sheat Catalyst IT Ltd. ✆ +64 4 803 2204 GPG: 5FA7 4B49 1E4D CAA4 4C38 8505 77F5 B724 F871 3BDF
participants (5)
-
Fridolin SOMERS -
Mark Tompsett -
Paul A -
Robin Sheat -
Tomas Cohen Arazi