Hello again all, looking at speed issues there is one thing that i don't understand, and i feel some well versed developpers might have a better idea of what is happening. If you query zebra directly on the server, it's blazingly fast, no matter how big your database and how many results you get. (At least the difference is negligible, and we still get very low response times.) But if you do a search in Koha that brings 20 000 results, it's usualy 4 times faster (maybe more, sorry i don't have precise metrics here) than a search that brings up 1 000 000 results. Is it consistent with your experience? If it is, what would be the reason for this? My understanding is that the search code asks zebra for the first n records to display on the first page, and that facets are based on a number of result way below 20 000 anyway, so the total number of results shouldn't really make a difference. -- Gaetan Boisson Chef de projet bibliothécaire BibLibre 06 52 42 51 29 108 avenue Breteuil 13006 Marseille gaetan.boisson@biblibre.com
Gaetan, have a look at the metrics on bug 13665. 2015-05-27 11:45 GMT+01:00 Gaetan Boisson <gaetan.boisson@biblibre.com>:
Hello again all,
looking at speed issues there is one thing that i don't understand, and i feel some well versed developpers might have a better idea of what is happening.
If you query zebra directly on the server, it's blazingly fast, no matter how big your database and how many results you get. (At least the difference is negligible, and we still get very low response times.)
But if you do a search in Koha that brings 20 000 results, it's usualy 4 times faster (maybe more, sorry i don't have precise metrics here) than a search that brings up 1 000 000 results.
Is it consistent with your experience?
If it is, what would be the reason for this? My understanding is that the search code asks zebra for the first n records to display on the first page, and that facets are based on a number of result way below 20 000 anyway, so the total number of results shouldn't really make a difference.
-- Gaetan Boisson Chef de projet bibliothécaire BibLibre 06 52 42 51 29 108 avenue Breteuil 13006 Marseille gaetan.boisson@biblibre.com
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
Well as i said, the time is not the same depending on the number of results, but in both cases, the number of results is anyway much higher than the number of records taken into consideration for facets. Your investigation indicates that: In ZOOM->record, the time is spent in my $_rec = Net::Z3950::ZOOM::resultset_record($this->_rs(), $which); Maybe it's worth having a deeper look in this. Le 27/05/2015 12:56, Jonathan Druart a écrit :
Gaetan, have a look at the metrics on bug 13665.
2015-05-27 11:45 GMT+01:00 Gaetan Boisson <gaetan.boisson@biblibre.com>:
Hello again all,
looking at speed issues there is one thing that i don't understand, and i feel some well versed developpers might have a better idea of what is happening.
If you query zebra directly on the server, it's blazingly fast, no matter how big your database and how many results you get. (At least the difference is negligible, and we still get very low response times.)
But if you do a search in Koha that brings 20 000 results, it's usualy 4 times faster (maybe more, sorry i don't have precise metrics here) than a search that brings up 1 000 000 results.
Is it consistent with your experience?
If it is, what would be the reason for this? My understanding is that the search code asks zebra for the first n records to display on the first page, and that facets are based on a number of result way below 20 000 anyway, so the total number of results shouldn't really make a difference.
-- Gaetan Boisson Chef de projet bibliothécaire BibLibre 06 52 42 51 29 108 avenue Breteuil 13006 Marseille gaetan.boisson@biblibre.com
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Gaetan Boisson Chef de projet bibliothécaire BibLibre 06 52 42 51 29 108 avenue Breteuil 13006 Marseille gaetan.boisson@biblibre.com
At 08:29 PM 5/27/2015 +0200, Gaetan Boisson wrote:
Well as i said, the time is not the same depending on the number of results, but in both cases, the number of results is anyway much higher than the number of records taken into consideration for facets.
Your investigation indicates that:
In ZOOM->record, the time is spent in my $_rec = Net::Z3950::ZOOM::resultset_record($this->_rs(), $which);
Maybe it's worth having a deeper look in this.
I looked into this to some extent in January this year; facets in 3.18 <http://navalmarinearchive.com/z_koha/search_speed_data.html> appeared to be a limiting factor as it "swamped" one CPU core (and NYProf showed this to be from ZOOM - see <http://navalmarinearchive.com/z_koha/nytprof_318_s/index.html> Regards -- Paul
Le 27/05/2015 12:56, Jonathan Druart a écrit :
Gaetan, have a look at the metrics on bug 13665.
2015-05-27 11:45 GMT+01:00 Gaetan Boisson <gaetan.boisson@biblibre.com>:
Hello again all,
looking at speed issues there is one thing that i don't understand, and i feel some well versed developpers might have a better idea of what is happening.
If you query zebra directly on the server, it's blazingly fast, no matter how big your database and how many results you get. (At least the difference is negligible, and we still get very low response times.)
But if you do a search in Koha that brings 20 000 results, it's usualy 4 times faster (maybe more, sorry i don't have precise metrics here) than a search that brings up 1 000 000 results.
Is it consistent with your experience?
If it is, what would be the reason for this? My understanding is that the search code asks zebra for the first n records to display on the first page, and that facets are based on a number of result way below 20 000 anyway, so the total number of results shouldn't really make a difference.
-- Gaetan Boisson Chef de projet bibliothécaire BibLibre 06 52 42 51 29 108 avenue Breteuil 13006 Marseille gaetan.boisson@biblibre.com
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Gaetan Boisson Chef de projet bibliothécaire BibLibre 06 52 42 51 29 108 avenue Breteuil 13006 Marseille gaetan.boisson@biblibre.com
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
--- Maritime heritage and history, preservation and conservation, research and education through the written word and the arts. <http://NavalMarineArchive.com> and <http://UltraMarine.ca>
Could this mean one should not get records from Zebra but directly from database ? If getting the id of search results without getting the full record is possible. Le 27/05/2015 21:02, Paul A a écrit :
At 08:29 PM 5/27/2015 +0200, Gaetan Boisson wrote:
Well as i said, the time is not the same depending on the number of results, but in both cases, the number of results is anyway much higher than the number of records taken into consideration for facets.
Your investigation indicates that:
In ZOOM->record, the time is spent in my $_rec = Net::Z3950::ZOOM::resultset_record($this->_rs(), $which);
Maybe it's worth having a deeper look in this.
I looked into this to some extent in January this year; facets in 3.18 <http://navalmarinearchive.com/z_koha/search_speed_data.html> appeared to be a limiting factor as it "swamped" one CPU core (and NYProf showed this to be from ZOOM - see <http://navalmarinearchive.com/z_koha/nytprof_318_s/index.html>
Regards -- Paul
Le 27/05/2015 12:56, Jonathan Druart a écrit :
Gaetan, have a look at the metrics on bug 13665.
2015-05-27 11:45 GMT+01:00 Gaetan Boisson <gaetan.boisson@biblibre.com>:
Hello again all,
looking at speed issues there is one thing that i don't understand, and i feel some well versed developpers might have a better idea of what is happening.
If you query zebra directly on the server, it's blazingly fast, no matter how big your database and how many results you get. (At least the difference is negligible, and we still get very low response times.)
But if you do a search in Koha that brings 20 000 results, it's usualy 4 times faster (maybe more, sorry i don't have precise metrics here) than a search that brings up 1 000 000 results.
Is it consistent with your experience?
If it is, what would be the reason for this? My understanding is that the search code asks zebra for the first n records to display on the first page, and that facets are based on a number of result way below 20 000 anyway, so the total number of results shouldn't really make a difference.
-- Gaetan Boisson Chef de projet bibliothécaire BibLibre 06 52 42 51 29 108 avenue Breteuil 13006 Marseille gaetan.boisson@biblibre.com
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Gaetan Boisson Chef de projet bibliothécaire BibLibre 06 52 42 51 29 108 avenue Breteuil 13006 Marseille gaetan.boisson@biblibre.com
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
--- Maritime heritage and history, preservation and conservation, research and education through the written word and the arts. <http://NavalMarineArchive.com> and <http://UltraMarine.ca>
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Fridolin SOMERS Biblibre - Pôles support et système fridolin.somers@biblibre.com
El 28/5/2015 4:43 a. m., "Fridolin SOMERS" <fridolin.somers@biblibre.com> escribió:
Could this mean one should not get records from Zebra but directly from
database ? If getting the id of search results without getting the full record is possible. It is possible. But we should evaluate the trade-off from preparing the record for display too. Because for indexing we do lots of stuff to the record, stuff that should be done on rendering time with such a change.
Le 27/05/2015 21:02, Paul A a écrit :
At 08:29 PM 5/27/2015 +0200, Gaetan Boisson wrote:
Well as i said, the time is not the same depending on the number of results, but in both cases, the number of results is anyway much higher than the number of records taken into consideration for facets.
Your investigation indicates that:
In ZOOM->record, the time is spent in my $_rec = Net::Z3950::ZOOM::resultset_record($this->_rs(), $which);
Maybe it's worth having a deeper look in this.
I looked into this to some extent in January this year; facets in 3.18 <http://navalmarinearchive.com/z_koha/search_speed_data.html> appeared to be a limiting factor as it "swamped" one CPU core (and NYProf showed this to be from ZOOM - see <http://navalmarinearchive.com/z_koha/nytprof_318_s/index.html>
Regards -- Paul
Le 27/05/2015 12:56, Jonathan Druart a écrit :
Gaetan, have a look at the metrics on bug 13665.
2015-05-27 11:45 GMT+01:00 Gaetan Boisson <gaetan.boisson@biblibre.com
:
Hello again all,
looking at speed issues there is one thing that i don't understand, and i feel some well versed developpers might have a better idea of what is happening.
If you query zebra directly on the server, it's blazingly fast, no matter how big your database and how many results you get. (At least the difference is negligible, and we still get very low response times.)
But if you do a search in Koha that brings 20 000 results, it's usualy 4 times faster (maybe more, sorry i don't have precise metrics here) than a search that brings up 1 000 000 results.
Is it consistent with your experience?
If it is, what would be the reason for this? My understanding is that the search code asks zebra for the first n records to display on the first page, and that facets are based on a number of result way below 20 000 anyway, so the total number of results shouldn't really make a difference.
-- Gaetan Boisson Chef de projet bibliothécaire BibLibre 06 52 42 51 29 108 avenue Breteuil 13006 Marseille gaetan.boisson@biblibre.com
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Gaetan Boisson Chef de projet bibliothécaire BibLibre 06 52 42 51 29 108 avenue Breteuil 13006 Marseille gaetan.boisson@biblibre.com
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
--- Maritime heritage and history, preservation and conservation, research and education through the written word and the arts. <http://NavalMarineArchive.com> and <http://UltraMarine.ca>
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Fridolin SOMERS Biblibre - Pôles support et système fridolin.somers@biblibre.com
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
Le 28/05/2015 12:52, Tomas Cohen Arazi a écrit :
El 28/5/2015 4:43 a. m., "Fridolin SOMERS" <fridolin.somers@biblibre.com> escribió:
Could this mean one should not get records from Zebra but directly from
database ? If getting the id of search results without getting the full record is possible.
It is possible. But we should evaluate the trade-off from preparing the record for display too. Because for indexing we do lots of stuff to the record, stuff that should be done on rendering time with such a change. What changes ? For detail page, the record comes from database so it should not be different for search results page.
Le 27/05/2015 21:02, Paul A a écrit :
At 08:29 PM 5/27/2015 +0200, Gaetan Boisson wrote:
Well as i said, the time is not the same depending on the number of results, but in both cases, the number of results is anyway much higher than the number of records taken into consideration for facets.
Your investigation indicates that:
In ZOOM->record, the time is spent in my $_rec = Net::Z3950::ZOOM::resultset_record($this->_rs(), $which);
Maybe it's worth having a deeper look in this.
I looked into this to some extent in January this year; facets in 3.18 <http://navalmarinearchive.com/z_koha/search_speed_data.html> appeared to be a limiting factor as it "swamped" one CPU core (and NYProf showed this to be from ZOOM - see <http://navalmarinearchive.com/z_koha/nytprof_318_s/index.html>
Regards -- Paul
Le 27/05/2015 12:56, Jonathan Druart a écrit :
Gaetan, have a look at the metrics on bug 13665.
2015-05-27 11:45 GMT+01:00 Gaetan Boisson <gaetan.boisson@biblibre.com
:
Hello again all,
looking at speed issues there is one thing that i don't understand, and i feel some well versed developpers might have a better idea of what is happening.
If you query zebra directly on the server, it's blazingly fast, no matter how big your database and how many results you get. (At least the difference is negligible, and we still get very low response times.)
But if you do a search in Koha that brings 20 000 results, it's usualy 4 times faster (maybe more, sorry i don't have precise metrics here) than a search that brings up 1 000 000 results.
Is it consistent with your experience?
If it is, what would be the reason for this? My understanding is that the search code asks zebra for the first n records to display on the first page, and that facets are based on a number of result way below 20 000 anyway, so the total number of results shouldn't really make a difference.
-- Gaetan Boisson Chef de projet bibliothécaire BibLibre 06 52 42 51 29 108 avenue Breteuil 13006 Marseille gaetan.boisson@biblibre.com
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Gaetan Boisson Chef de projet bibliothécaire BibLibre 06 52 42 51 29 108 avenue Breteuil 13006 Marseille gaetan.boisson@biblibre.com
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
--- Maritime heritage and history, preservation and conservation, research and education through the written word and the arts. <http://NavalMarineArchive.com> and <http://UltraMarine.ca>
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Fridolin SOMERS Biblibre - Pôles support et système fridolin.somers@biblibre.com
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Fridolin SOMERS Biblibre - Pôles support et système fridolin.somers@biblibre.com
Dear Fridolin, I see C4/Context.pm dbh no mysql_use_result options, We can try it. but it maybe modify koha SQL statement. http://search.cpan.org/~capttofu/DBD-mysql-4.011/lib/DBD/mysql.pm This attribute forces the driver to use mysql_use_result rather than mysql_store_result. The former is faster and less memory consuming, but tends to block other processes. (That's why mysql_store_result is the default.) BR. 龍山,
Fridolin SOMERS <fridolin.somers@biblibre.com> 於 2015/5/28 (週四) 11:24 PM 寫道﹕
Le 28/05/2015 12:52, Tomas Cohen Arazi a écrit :
El 28/5/2015 4:43 a. m., "Fridolin SOMERS" <fridolin.somers@biblibre.com> escribió:
Could this mean one should not get records from Zebra but directly from
database ? If getting the id of search results without getting the full record is possible.
It is possible. But we should evaluate the trade-off from preparing the record for display too. Because for indexing we do lots of stuff to the record, stuff that should be done on rendering time with such a change. What changes ? For detail page, the record comes from database so it should not be different for search results page.
Le 27/05/2015 21:02, Paul A a écrit :
At 08:29 PM 5/27/2015 +0200, Gaetan Boisson wrote:
Well as i said, the time is not the same depending on the
results, but in both cases, the number of results is anyway much higher than the number of records taken into consideration for facets.
Your investigation indicates that:
In ZOOM->record, the time is spent in my $_rec = Net::Z3950::ZOOM::resultset_record($this->_rs(), $which);
Maybe it's worth having a deeper look in this.
I looked into this to some extent in January this year; facets in 3.18 <http://navalmarinearchive.com/z_koha/search_speed_data.html> appeared to be a limiting factor as it "swamped" one CPU core (and NYProf showed this to be from ZOOM - see
<http://navalmarinearchive.com/z_koha/nytprof_318_s/index.html>
Regards -- Paul
Le 27/05/2015 12:56, Jonathan Druart a écrit :
Gaetan, have a look at the metrics on bug 13665.
2015-05-27 11:45 GMT+01:00 Gaetan Boisson
<gaetan.boisson@biblibre.com :
> > Hello again all, > > looking at speed issues there is one thing that i don't understand, > and i > feel some well versed developpers might have a better idea of what is > happening. > > If you query zebra directly on the server, it's blazingly fast, no > matter > how big your database and how many results you get. (At least the > difference > is negligible, and we still get very low response times.) > > But if you do a search in Koha that brings 20 000 results, it's > usualy 4 > times faster (maybe more, sorry i don't have
number of precise metrics here)
> than a > search that brings up 1 000 000 results. > > Is it consistent with your experience? > > If it is, what would be the reason for this? My understanding is > that the > search code asks zebra for the first n records to display on the > first page, > and that facets are based on a number of result way below 20 000 > anyway, so > the total number of results shouldn't really make a difference. > > -- > Gaetan Boisson > Chef de projet bibliothécaire > BibLibre > 06 52 42 51 29 > 108 avenue Breteuil 13006 Marseille > gaetan.boisson@biblibre.com > > _______________________________________________ > Koha-devel mailing list > Koha-devel@lists.koha-community.org > http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > website : http://www.koha-community.org/ > git : http://git.koha-community.org/ > bugs : http://bugs.koha-community.org/
-- Gaetan Boisson Chef de projet bibliothécaire BibLibre 06 52 42 51 29 108 avenue Breteuil 13006 Marseille gaetan.boisson@biblibre.com
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
--- Maritime heritage and history, preservation and conservation, research and education through the written word and the arts. <http://NavalMarineArchive.com> and <http://UltraMarine.ca>
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Fridolin SOMERS Biblibre - Pôles support et système
fridolin.somers@biblibre.com
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Fridolin SOMERS Biblibre - Pôles support et système fridolin.somers@biblibre.com _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
participants (6)
-
Fridolin SOMERS -
Gaetan Boisson -
Jonathan Druart -
long_sam.tw -
Paul A -
Tomas Cohen Arazi