Re: [Koha-devel] What's on in Koha devel #3 (RESTful API)
= RESTful API = I also would like to know who is willing to maintain this code? For instance on bug 16330 (REST API: add routes to add, update and delete patrons) the delete route is calling # check if loans, reservations, debarrment, etc. before deletion! my $res = $patron->delete So no check is done from the API. But if bug 16907 (Move DelMember and HandleDelBorrower to Koha::Patron) is pushed, $patron->delete will take care of the holds, etc. My question is simple: are we expecting from devs, signoffers and > QAers to update and maintain these routes? Or a dedicated person will update them before each release?
Just formally, I would say that if the community agreed to push this code, the community should maintain it too. We should keep the code base in a consistent state. Similarly, we should prevent redundant code: the checks for deleting a patron in one place, etc. If we are on a road now creating a lot of redundant code between Koha/REST and Koha/Objects, we should re-evaluate first and not just keep pushing more code. Is this mainly a communication issue? Which goals do we want to achieve? Should our scripts go via REST some day and should all business logic remain in Objects? Etc. etc. Marcel
El lun., 15 ago. 2016 a las 4:28, Marcel de Rooy (<M.de.Rooy@rijksmuseum.nl>) escribió:
= RESTful API = I also would like to know who is willing to maintain this code? For instance on bug 16330 (REST API: add routes to add, update and delete patrons) the delete route is calling # check if loans, reservations, debarrment, etc. before deletion! my $res = $patron->delete So no check is done from the API. But if bug 16907 (Move DelMember and HandleDelBorrower to Koha::Patron) is pushed, $patron->delete will take care of the holds, etc. My question is simple: are we expecting from devs, signoffers and > QAers to update and maintain these routes? Or a dedicated person will update them before each release?
Should our scripts go via REST some day and should all business logic remain in Objects? Etc. etc.
Exactly! What Jonathan noticed, is that no one realized there was a bug on the REST api, this should be reported, and fixed. The more we use the API the more bugs we will find, and the more we will enhance it. I think Marseille will be a good place to work on this. Regards -- Tomás Cohen Arazi Theke Solutions (https://theke.io <http://theke.io/>) ✆ +54 9351 3513384 GPG: B2F3C15F
Am 15.08.2016 um 14:54 schrieb Tomas Cohen Arazi:
El lun., 15 ago. 2016 a las 4:28, Marcel de Rooy (<M.de.Rooy@rijksmuseum.nl <mailto:M.de.Rooy@rijksmuseum.nl>>) escribió:
> = RESTful API = > I also would like to know who is willing to maintain this code? For instance on bug 16330 (REST API: add routes to add, update and delete > patrons) the delete route is calling > # check if loans, reservations, debarrment, etc. before deletion! > my $res = $patron->delete > So no check is done from the API. > But if bug 16907 (Move DelMember and HandleDelBorrower to > Koha::Patron) is pushed, $patron->delete will take care of the holds, etc. My question is simple: are we expecting from devs, signoffers and > QAers to update and maintain these routes? > Or a dedicated person will update them before each release?
Should our scripts go via REST some day and should all business logic remain in Objects? Etc. etc.
Exactly! What Jonathan noticed, is that no one realized there was a bug on the REST api, this should be reported, and fixed. The more we use the API the more bugs we will find, and the more we will enhance it.
I think Marseille will be a good place to work on this.
Regards
Maybe Marseille will also be a good place for a 'beginners tutorial'? I'd like to learn more about the RESTful API and how to use/test it in order to help with improving it :) Katrin
In my mind, we should be moving all appropriate logic to Koha/Objects with Koha/REST being just an api wrapper around the appropriate object. That will keep us from duplicating code and reduce bugs. Kyle <https://secure2.convio.net/cffh/site/Donation2?df_id=1395&FR_ID=4715&PROXY_ID=2706639&PROXY_TYPE=20&1395.donation=form1&s_src=CHORUS&s_subsrc=CHAADOEB> http://www.kylehall.info ByWater Solutions ( http://bywatersolutions.com ) Meadville Public Library ( http://www.meadvillelibrary.org ) Crawford County Federated Library System ( http://www.ccfls.org ) On Mon, Aug 15, 2016 at 3:28 AM, Marcel de Rooy <M.de.Rooy@rijksmuseum.nl> wrote:
= RESTful API = I also would like to know who is willing to maintain this code? For instance on bug 16330 (REST API: add routes to add, update and delete patrons) the delete route is calling # check if loans, reservations, debarrment, etc. before deletion! my $res = $patron->delete So no check is done from the API. But if bug 16907 (Move DelMember and HandleDelBorrower to Koha::Patron) is pushed, $patron->delete will take care of the holds, etc. My question is simple: are we expecting from devs, signoffers and > QAers to update and maintain these routes? Or a dedicated person will update them before each release?
Just formally, I would say that if the community agreed to push this code, the community should maintain it too. We should keep the code base in a consistent state. Similarly, we should prevent redundant code: the checks for deleting a patron in one place, etc.
If we are on a road now creating a lot of redundant code between Koha/REST and Koha/Objects, we should re-evaluate first and not just keep pushing more code. Is this mainly a communication issue? Which goals do we want to achieve? Should our scripts go via REST some day and should all business logic remain in Objects? Etc. etc.
Marcel
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
+1 The REST API is just a public interface. It should use the same internal APIs as the rest of Koha. David Cook Systems Librarian Prosentient Systems 72/330 Wattle St Ultimo, NSW 2007 Australia Office: 02 9212 0899 Direct: 02 8005 0595 From: koha-devel-bounces@lists.koha-community.org [mailto:koha-devel-bounces@lists.koha-community.org] On Behalf Of Kyle Hall Sent: Wednesday, 17 August 2016 4:31 AM To: Marcel de Rooy <M.de.Rooy@rijksmuseum.nl> Cc: koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] What's on in Koha devel #3 (RESTful API) In my mind, we should be moving all appropriate logic to Koha/Objects with Koha/REST being just an api wrapper around the appropriate object. That will keep us from duplicating code and reduce bugs. Kyle <https://secure2.convio.net/cffh/site/Donation2?df_id=1395&FR_ID=4715&PROXY_ID=2706639&PROXY_TYPE=20&1395.donation=form1&s_src=CHORUS&s_subsrc=CHAADOEB> http://www.kylehall.info ByWater Solutions ( http://bywatersolutions.com ) Meadville Public Library ( http://www.meadvillelibrary.org ) Crawford County Federated Library System ( http://www.ccfls.org ) On Mon, Aug 15, 2016 at 3:28 AM, Marcel de Rooy <M.de.Rooy@rijksmuseum.nl <mailto:M.de.Rooy@rijksmuseum.nl> > wrote:
= RESTful API = I also would like to know who is willing to maintain this code? For instance on bug 16330 (REST API: add routes to add, update and delete patrons) the delete route is calling # check if loans, reservations, debarrment, etc. before deletion! my $res = $patron->delete So no check is done from the API. But if bug 16907 (Move DelMember and HandleDelBorrower to Koha::Patron) is pushed, $patron->delete will take care of the holds, etc. My question is simple: are we expecting from devs, signoffers and > QAers to update and maintain these routes? Or a dedicated person will update them before each release?
Just formally, I would say that if the community agreed to push this code, the community should maintain it too. We should keep the code base in a consistent state. Similarly, we should prevent redundant code: the checks for deleting a patron in one place, etc. If we are on a road now creating a lot of redundant code between Koha/REST and Koha/Objects, we should re-evaluate first and not just keep pushing more code. Is this mainly a communication issue? Which goals do we want to achieve? Should our scripts go via REST some day and should all business logic remain in Objects? Etc. etc. Marcel _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
Hello! We're trying to find the correction combination. We're new on Jessie, so we still have some tweaking needed... - By default, we get zebra 2.00.59 installed on Jessie through the packages. - On 16.05, we get some very bad results in the search when the itemtype contains an hyphen (-), like 'A-DOC'. - So we installed zebra 2.00.62. This fixes the search... - But now we do not have facets. - So we set <use_zebra_facets>0</use_zebra_facets> - And now we have facets. But this feels... wrong? My dummy question: what is the supposedly correct version of Zebra on Jessie ? And we're we correct in setting the config to 0 ? Thanks Blou
Greetings, Similar problem. I hope someone has a better solution than setting it to 0. GPML, Mark Tompsett -----Original Message----- From: Philippe Blouin Sent: Monday, August 22, 2016 2:40 PM To: koha-devel@lists.koha-community.org Subject: [Koha-devel] 16.05, zebra and jessie Hello! We're trying to find the correction combination. We're new on Jessie, so we still have some tweaking needed... - By default, we get zebra 2.00.59 installed on Jessie through the packages. - On 16.05, we get some very bad results in the search when the itemtype contains an hyphen (-), like 'A-DOC'. - So we installed zebra 2.00.62. This fixes the search... - But now we do not have facets. - So we set <use_zebra_facets>0</use_zebra_facets> - And now we have facets. But this feels... wrong? My dummy question: what is the supposedly correct version of Zebra on Jessie ? And we're we correct in setting the config to 0 ? Thanks Blou _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
I haven't run into the issue with the dashes in idzebra-2.0 2.0.59, but I have run into this, when using ICU-Chains: Bug 16581 : ICU tokenization bug in idzebra-2.0 2.0.59-1 URL : https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16581 Priority : P5 - low Urgency : enhancement Status : NEW I also know that when use_zebra_facets was first introduced, it was *very* slow -- I can't find any bugs about that though. It's possible that it got so slow under idzebra-2.0 2.61 that the searches are timing out. It should be possible to set the logging for zebra so that you can see the PQF queries: See Bug 15714 : Remove zebra.log from debian scripts and add optional log levels URL : https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15714 Priority : P5 - low Urgency : enhancement Status : RESOLVED For setting the log levels And http://koha.1045719.n5.nabble.com/Improving-Zebra-logging-td5861827.html For a general discussion of how to use them. ... This should give you some idea of what's failing, both in terms of the dashes in 2.0.59 and the non-functional zebra facets under 2.0.61. My general feeling is that 2.0.59 is irredeemably broken by bug 16581, and we need at least 2.0.60, but I don't have any experience with zebra facets. --Barton On Mon, Aug 22, 2016 at 2:49 PM, Mark Tompsett <mtompset@hotmail.com> wrote:
Greetings,
Similar problem. I hope someone has a better solution than setting it to 0.
GPML, Mark Tompsett
-----Original Message----- From: Philippe Blouin Sent: Monday, August 22, 2016 2:40 PM To: koha-devel@lists.koha-community.org Subject: [Koha-devel] 16.05, zebra and jessie
Hello!
We're trying to find the correction combination. We're new on Jessie, so we still have some tweaking needed...
- By default, we get zebra 2.00.59 installed on Jessie through the packages. - On 16.05, we get some very bad results in the search when the itemtype contains an hyphen (-), like 'A-DOC'. - So we installed zebra 2.00.62. This fixes the search... - But now we do not have facets. - So we set <use_zebra_facets>0</use_zebra_facets> - And now we have facets. But this feels... wrong?
My dummy question: what is the supposedly correct version of Zebra on Jessie ? And we're we correct in setting the config to 0 ?
Thanks Blou _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
See bug 13665 - Retrieve facets from zebra is slow To understand why and when use_zebra_facet=1 is slow 2016-08-22 21:31 GMT+01:00 Barton Chittenden <barton@bywatersolutions.com>:
I haven't run into the issue with the dashes in idzebra-2.0 2.0.59, but I have run into this, when using ICU-Chains:
Bug 16581 : ICU tokenization bug in idzebra-2.0 2.0.59-1 URL : https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16581 Priority : P5 - low Urgency : enhancement Status : NEW
I also know that when use_zebra_facets was first introduced, it was *very* slow -- I can't find any bugs about that though. It's possible that it got so slow under idzebra-2.0 2.61 that the searches are timing out.
It should be possible to set the logging for zebra so that you can see the PQF queries:
See
Bug 15714 : Remove zebra.log from debian scripts and add optional log levels URL : https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15714 Priority : P5 - low Urgency : enhancement Status : RESOLVED
For setting the log levels
And http://koha.1045719.n5.nabble.com/Improving-Zebra-logging-td5861827.html
For a general discussion of how to use them.
... This should give you some idea of what's failing, both in terms of the dashes in 2.0.59 and the non-functional zebra facets under 2.0.61.
My general feeling is that 2.0.59 is irredeemably broken by bug 16581, and we need at least 2.0.60, but I don't have any experience with zebra facets.
--Barton
On Mon, Aug 22, 2016 at 2:49 PM, Mark Tompsett <mtompset@hotmail.com> wrote:
Greetings,
Similar problem. I hope someone has a better solution than setting it to 0.
GPML, Mark Tompsett
-----Original Message----- From: Philippe Blouin Sent: Monday, August 22, 2016 2:40 PM To: koha-devel@lists.koha-community.org Subject: [Koha-devel] 16.05, zebra and jessie
Hello!
We're trying to find the correction combination. We're new on Jessie, so we still have some tweaking needed...
- By default, we get zebra 2.00.59 installed on Jessie through the packages. - On 16.05, we get some very bad results in the search when the itemtype contains an hyphen (-), like 'A-DOC'. - So we installed zebra 2.00.62. This fixes the search... - But now we do not have facets. - So we set <use_zebra_facets>0</use_zebra_facets> - And now we have facets. But this feels... wrong?
My dummy question: what is the supposedly correct version of Zebra on Jessie ? And we're we correct in setting the config to 0 ?
Thanks Blou _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
Zebra tends to be I/O bound -- we've seen it write enormous .zrs files to disk (~16G/query on large libraries). Bug 13665 mentions that searches could be taking upwards of 40 seconds to complete -- I think that we've seen searches time out and return no results at about 1 minute. Is it possible to tune Zebra's space/time optimizations in any way so that it doesn't write such large files to disk? On Tue, Aug 23, 2016 at 5:38 AM, Jonathan Druart < jonathan.druart@bugs.koha-community.org> wrote:
See bug 13665 - Retrieve facets from zebra is slow To understand why and when use_zebra_facet=1 is slow
: I haven't run into the issue with the dashes in idzebra-2.0 2.0.59, but I have run into this, when using ICU-Chains:
Bug 16581 : ICU tokenization bug in idzebra-2.0 2.0.59-1 URL : https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id= 16581 Priority : P5 - low Urgency : enhancement Status : NEW
I also know that when use_zebra_facets was first introduced, it was *very* slow -- I can't find any bugs about that though. It's possible that it got so slow under idzebra-2.0 2.61 that the searches are timing out.
It should be possible to set the logging for zebra so that you can see
PQF queries:
See
Bug 15714 : Remove zebra.log from debian scripts and add optional log levels URL : https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id= 15714 Priority : P5 - low Urgency : enhancement Status : RESOLVED
For setting the log levels
And http://koha.1045719.n5.nabble.com/Improving-Zebra-logging-
2016-08-22 21:31 GMT+01:00 Barton Chittenden <barton@bywatersolutions.com the td5861827.html
For a general discussion of how to use them.
... This should give you some idea of what's failing, both in terms of
the
dashes in 2.0.59 and the non-functional zebra facets under 2.0.61.
My general feeling is that 2.0.59 is irredeemably broken by bug 16581, and we need at least 2.0.60, but I don't have any experience with zebra facets.
--Barton
On Mon, Aug 22, 2016 at 2:49 PM, Mark Tompsett <mtompset@hotmail.com> wrote:
Greetings,
Similar problem. I hope someone has a better solution than setting it to 0.
GPML, Mark Tompsett
-----Original Message----- From: Philippe Blouin Sent: Monday, August 22, 2016 2:40 PM To: koha-devel@lists.koha-community.org Subject: [Koha-devel] 16.05, zebra and jessie
Hello!
We're trying to find the correction combination. We're new on Jessie, so we still have some tweaking needed...
- By default, we get zebra 2.00.59 installed on Jessie through the packages. - On 16.05, we get some very bad results in the search when the itemtype contains an hyphen (-), like 'A-DOC'. - So we installed zebra 2.00.62. This fixes the search... - But now we do not have facets. - So we set <use_zebra_facets>0</use_zebra_facets> - And now we have facets. But this feels... wrong?
My dummy question: what is the supposedly correct version of Zebra on Jessie ? And we're we correct in setting the config to 0 ?
Thanks Blou _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
I have seen use_zebra_facets=1 cause no facets rendered when GRS-1 configuration files are kept during upgrades up to where GRS-1 got deprecated (3.20?). Is it the case? What does the About > System information page says about your config? The slowliness is not in zebra per se, but in the way we retrieve the facets from it (so Koha/Perl side). We retrieve each facet at a time instead of fetching them all in one call. And they come in XML format, so need to be parsed. So, if anyone is willing to improve it, just need to optimize this function (read the TODO): sub _get_facet_from_result_set { my $facet_idx = shift; my $rs = shift; my $sep = shift; my $internal_sep = '<*>'; my $facetMaxCount = C4::Context->preference('FacetMaxCount') // 20; return if ( ! defined $facet_idx || ! defined $rs ); # zebra's facet element, untokenized index my $facet_element = 'zebra::facet::' . $facet_idx . ':0:' . $facetMaxCount; # configure zebra results for retrieving the desired facet $rs->option( elementSetName => $facet_element ); # get the facet record from result set my $facet = $rs->record( 0 )->raw; # if the facet has no restuls... return if !defined $facet; # TODO: benchmark DOM vs. SAX performance my $facet_dom = XML::LibXML->load_xml( string => ($facet) ); my @terms = $facet_dom->getElementsByTagName('term'); return if ! @terms; my $facets = {}; foreach my $term ( @terms ) { my $facet_value = $term->textContent; $facet_value =~ s/\Q$internal_sep\E/$sep/ if defined $sep; $facets->{ $facet_value } = $term->getAttribute( 'occur' ); } return $facets; } Another option would be to make _get_facets_from_zebra build the element set containing all facets so they are read in one call (comma-separate all elements). The problem is that Zebra returns zero if one of the elements is empty. Jared proposed to create a ghost record with all facet fields. I didn't manage to make it work. Another option is to patch Zebra. I started that, but abandoned once the ES code got pushed. So, if use_zebra_facets=0 is good enough, maybe it should be recommended. Problem is it is not a real facet, but the sole extraction of the fields from the first x records. As I said, it could be good enough anyway. Regards El mar., 23 ago. 2016 a las 10:21, Barton Chittenden (< barton@bywatersolutions.com>) escribió:
Zebra tends to be I/O bound -- we've seen it write enormous .zrs files to disk (~16G/query on large libraries). Bug 13665 mentions that searches could be taking upwards of 40 seconds to complete -- I think that we've seen searches time out and return no results at about 1 minute.
Is it possible to tune Zebra's space/time optimizations in any way so that it doesn't write such large files to disk?
On Tue, Aug 23, 2016 at 5:38 AM, Jonathan Druart < jonathan.druart@bugs.koha-community.org> wrote:
See bug 13665 - Retrieve facets from zebra is slow To understand why and when use_zebra_facet=1 is slow
: I haven't run into the issue with the dashes in idzebra-2.0 2.0.59, but I have run into this, when using ICU-Chains:
Bug 16581 : ICU tokenization bug in idzebra-2.0 2.0.59-1 URL : https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16581 Priority : P5 - low Urgency : enhancement Status : NEW
I also know that when use_zebra_facets was first introduced, it was *very* slow -- I can't find any bugs about that though. It's possible that it got so slow under idzebra-2.0 2.61 that the searches are timing out.
It should be possible to set the logging for zebra so that you can see
PQF queries:
See
Bug 15714 : Remove zebra.log from debian scripts and add optional log levels URL : https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15714 Priority : P5 - low Urgency : enhancement Status : RESOLVED
For setting the log levels
And http://koha.1045719.n5.nabble.com/Improving-Zebra-logging-td5861827.html
For a general discussion of how to use them.
... This should give you some idea of what's failing, both in terms of
2016-08-22 21:31 GMT+01:00 Barton Chittenden <barton@bywatersolutions.com the the
dashes in 2.0.59 and the non-functional zebra facets under 2.0.61.
My general feeling is that 2.0.59 is irredeemably broken by bug 16581, and we need at least 2.0.60, but I don't have any experience with zebra facets.
--Barton
On Mon, Aug 22, 2016 at 2:49 PM, Mark Tompsett <mtompset@hotmail.com> wrote:
Greetings,
Similar problem. I hope someone has a better solution than setting it
to
0.
GPML, Mark Tompsett
-----Original Message----- From: Philippe Blouin Sent: Monday, August 22, 2016 2:40 PM To: koha-devel@lists.koha-community.org Subject: [Koha-devel] 16.05, zebra and jessie
Hello!
We're trying to find the correction combination. We're new on Jessie, so we still have some tweaking needed...
- By default, we get zebra 2.00.59 installed on Jessie through the packages. - On 16.05, we get some very bad results in the search when the itemtype contains an hyphen (-), like 'A-DOC'. - So we installed zebra 2.00.62. This fixes the search... - But now we do not have facets. - So we set <use_zebra_facets>0</use_zebra_facets> - And now we have facets. But this feels... wrong?
My dummy question: what is the supposedly correct version of Zebra on Jessie ? And we're we correct in setting the config to 0 ?
Thanks Blou _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Tomás Cohen Arazi Theke Solutions (https://theke.io <http://theke.io/>) ✆ +54 9351 3513384 GPG: B2F3C15F
If I had more time, I’d patch Zebra… It’s been a while since I looked at this. Is the problem that no facets are returned if one of them would return zero, or that no facets are returned if one of the fields specified in the facet browse is an empty index/un-indexed field in Zebra? If it’s the latter, I wonder if using the Zebra specific search extension “14” in the main query would help… (see “Specifies whether un-indexed fields should be ignored. A zero value (default) throws a diagnostic when an un-indexed field is specified. A non-zero value makes it return 0 hits.” <http://www.indexdata.com/zebra/doc/zebra.pdf> http://www.indexdata.com/zebra/doc/zebra.pdf). I suppose Adam at IndexData has been busy with the FOLIO project, so I doubt he has time to work on Zebra these days, even if we did have a patch available. Is ElasticSearch usable with Koha at this point? I heard a lot in 2015, but after Robin left I haven’t heard a word other than rumours that the patches had been pushed? David Cook Systems Librarian Prosentient Systems 72/330 Wattle St Ultimo, NSW 2007 Australia Office: 02 9212 0899 Direct: 02 8005 0595 From: koha-devel-bounces@lists.koha-community.org [mailto:koha-devel-bounces@lists.koha-community.org] On Behalf Of Tomas Cohen Arazi Sent: Tuesday, 23 August 2016 11:42 PM To: Barton Chittenden <barton@bywatersolutions.com>; Jonathan Druart <jonathan.druart@bugs.koha-community.org> Cc: koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] 16.05, zebra and jessie I have seen use_zebra_facets=1 cause no facets rendered when GRS-1 configuration files are kept during upgrades up to where GRS-1 got deprecated (3.20?). Is it the case? What does the About > System information page says about your config? The slowliness is not in zebra per se, but in the way we retrieve the facets from it (so Koha/Perl side). We retrieve each facet at a time instead of fetching them all in one call. And they come in XML format, so need to be parsed. So, if anyone is willing to improve it, just need to optimize this function (read the TODO): sub _get_facet_from_result_set { my $facet_idx = shift; my $rs = shift; my $sep = shift; my $internal_sep = '<*>'; my $facetMaxCount = C4::Context->preference('FacetMaxCount') // 20; return if ( ! defined $facet_idx || ! defined $rs ); # zebra's facet element, untokenized index my $facet_element = 'zebra::facet::' . $facet_idx . ':0:' . $facetMaxCount; # configure zebra results for retrieving the desired facet $rs->option( elementSetName => $facet_element ); # get the facet record from result set my $facet = $rs->record( 0 )->raw; # if the facet has no restuls... return if !defined $facet; # TODO: benchmark DOM vs. SAX performance my $facet_dom = XML::LibXML->load_xml( string => ($facet) ); my @terms = $facet_dom->getElementsByTagName('term'); return if ! @terms; my $facets = {}; foreach my $term ( @terms ) { my $facet_value = $term->textContent; $facet_value =~ s/\Q$internal_sep\E/$sep/ if defined $sep; $facets->{ $facet_value } = $term->getAttribute( 'occur' ); } return $facets; } Another option would be to make _get_facets_from_zebra build the element set containing all facets so they are read in one call (comma-separate all elements). The problem is that Zebra returns zero if one of the elements is empty. Jared proposed to create a ghost record with all facet fields. I didn't manage to make it work. Another option is to patch Zebra. I started that, but abandoned once the ES code got pushed. So, if use_zebra_facets=0 is good enough, maybe it should be recommended. Problem is it is not a real facet, but the sole extraction of the fields from the first x records. As I said, it could be good enough anyway. Regards El mar., 23 ago. 2016 a las 10:21, Barton Chittenden (<barton@bywatersolutions.com <mailto:barton@bywatersolutions.com> >) escribió: Zebra tends to be I/O bound -- we've seen it write enormous .zrs files to disk (~16G/query on large libraries). Bug 13665 mentions that searches could be taking upwards of 40 seconds to complete -- I think that we've seen searches time out and return no results at about 1 minute. Is it possible to tune Zebra's space/time optimizations in any way so that it doesn't write such large files to disk? On Tue, Aug 23, 2016 at 5:38 AM, Jonathan Druart <jonathan.druart@bugs.koha-community.org <mailto:jonathan.druart@bugs.koha-community.org> > wrote: See bug 13665 - Retrieve facets from zebra is slow To understand why and when use_zebra_facet=1 is slow 2016-08-22 21:31 GMT+01:00 Barton Chittenden <barton@bywatersolutions.com <mailto:barton@bywatersolutions.com> >:
I haven't run into the issue with the dashes in idzebra-2.0 2.0.59, but I have run into this, when using ICU-Chains:
Bug 16581 : ICU tokenization bug in idzebra-2.0 2.0.59-1 URL : https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16581 Priority : P5 - low Urgency : enhancement Status : NEW
I also know that when use_zebra_facets was first introduced, it was *very* slow -- I can't find any bugs about that though. It's possible that it got so slow under idzebra-2.0 2.61 that the searches are timing out.
It should be possible to set the logging for zebra so that you can see the PQF queries:
See
Bug 15714 : Remove zebra.log from debian scripts and add optional log levels URL : https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15714 Priority : P5 - low Urgency : enhancement Status : RESOLVED
For setting the log levels
And http://koha.1045719.n5.nabble.com/Improving-Zebra-logging-td5861827.html
For a general discussion of how to use them.
... This should give you some idea of what's failing, both in terms of the dashes in 2.0.59 and the non-functional zebra facets under 2.0.61.
My general feeling is that 2.0.59 is irredeemably broken by bug 16581, and we need at least 2.0.60, but I don't have any experience with zebra facets.
--Barton
On Mon, Aug 22, 2016 at 2:49 PM, Mark Tompsett <mtompset@hotmail.com <mailto:mtompset@hotmail.com> > wrote:
Greetings,
Similar problem. I hope someone has a better solution than setting it to 0.
GPML, Mark Tompsett
-----Original Message----- From: Philippe Blouin Sent: Monday, August 22, 2016 2:40 PM To: koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org> Subject: [Koha-devel] 16.05, zebra and jessie
Hello!
We're trying to find the correction combination. We're new on Jessie, so we still have some tweaking needed...
- By default, we get zebra 2.00.59 installed on Jessie through the packages. - On 16.05, we get some very bad results in the search when the itemtype contains an hyphen (-), like 'A-DOC'. - So we installed zebra 2.00.62. This fixes the search... - But now we do not have facets. - So we set <use_zebra_facets>0</use_zebra_facets> - And now we have facets. But this feels... wrong?
My dummy question: what is the supposedly correct version of Zebra on Jessie ? And we're we correct in setting the config to 0 ?
Thanks Blou _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/ -- Tomás Cohen Arazi Theke Solutions (https://theke.io <http://theke.io/> ) ✆ +54 9351 3513384 GPG: B2F3C15F
* David Cook (dcook@prosentient.com.au) wrote:
I suppose Adam at IndexData has been busy with the FOLIO project, so I doubt he has time to work on Zebra these days, even if we did have a patch available.
Is ElasticSearch usable with Koha at this point? I heard a lot in 2015, but after Robin left I haven’t heard a word other than rumours that the patches had been pushed?
You missed tons of words then :) Yes, it is in 16.05, marked expiremental, it works, mostly. But it will only get better with more people using and fixing it. The next task is to update the version it works with to Elastic 2. That isn't a huge amount of work, but everyone has their own priorities and a lot of us have to work on what users ask for (not what users need ;)) Chris
Subject: Re: [Koha-devel] 16.05, zebra and jessie
I have seen use_zebra_facets=1 cause no facets rendered when GRS-1 configuration files are kept during upgrades up to where GRS-1 got deprecated (3.20?). Is it the case? What does the About > System information page says about your config?
The slowliness is not in zebra per se, but in the way we retrieve the facets from it (so Koha/Perl side). We retrieve each facet at a time instead of fetching them all in one call. And they come in XML format, so need to be parsed. So, if anyone is willing to improve it, just need to optimize this function (read the TODO):
sub _get_facet_from_result_set {
my $facet_idx = shift;
my $rs = shift;
my $sep = shift;
my $internal_sep = '<*>';
my $facetMaxCount = C4::Context->preference('FacetMaxCount') // 20;
return if ( ! defined $facet_idx || ! defined $rs );
# zebra's facet element, untokenized index
my $facet_element = 'zebra::facet::' . $facet_idx . ':0:' . $facetMaxCount;
# configure zebra results for retrieving the desired facet
$rs->option( elementSetName => $facet_element );
# get the facet record from result set
my $facet = $rs->record( 0 )->raw;
# if the facet has no restuls...
return if !defined $facet;
# TODO: benchmark DOM vs. SAX performance
my $facet_dom = XML::LibXML->load_xml(
string => ($facet)
);
my @terms = $facet_dom->getElementsByTagName('term');
return if ! @terms;
my $facets = {};
foreach my $term ( @terms ) {
my $facet_value = $term->textContent;
$facet_value =~ s/\Q$internal_sep\E/$sep/ if defined $sep;
$facets->{ $facet_value } = $term->getAttribute( 'occur' );
}
return $facets;
}
Another option would be to make _get_facets_from_zebra build the element set containing all facets so they are read in one call (comma-separate all elements). The problem is that Zebra returns zero if one of the elements is empty. Jared proposed to create a ghost record with all facet fields. I didn't manage to make it work. Another option is to patch Zebra. I started that, but abandoned once the ES code got pushed.
So, if use_zebra_facets=0 is good enough, maybe it should be recommended. Problem is it is not a real facet, but the sole extraction of the fields from the first x records.
As I said, it could be good enough anyway.
Regards
El mar., 23 ago. 2016 a las 10:21, Barton Chittenden (< barton@bywatersolutions.com>) escribió:
Zebra tends to be I/O bound -- we've seen it write enormous .zrs files to disk (~16G/query on large libraries). Bug 13665 mentions that searches could be taking upwards of 40 seconds to complete -- I think that we've seen searches time out and return no results at about 1 minute.
Is it possible to tune Zebra's space/time optimizations in any way so that it doesn't write such large files to disk?
On Tue, Aug 23, 2016 at 5:38 AM, Jonathan Druart < jonathan.druart@bugs.koha-community.org> wrote:
See bug 13665 - Retrieve facets from zebra is slow To understand why and when use_zebra_facet=1 is slow
2016-08-22 21:31 GMT+01:00 Barton Chittenden < barton@bywatersolutions.com>: > I haven't run into the issue with the dashes in idzebra-2.0 2.0.59, but I > have run into this, when using ICU-Chains: > > Bug 16581 : ICU tokenization bug in idzebra-2.0 2.0.59-1 > URL : https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id =16581 > Priority : P5 - low > Urgency : enhancement > Status : NEW > > I also know that when use_zebra_facets was first introduced, it was *very* > slow -- I can't find any bugs about that though. It's possible that it got > so slow under idzebra-2.0 2.61 that the searches are timing out. > > It should be possible to set the logging for zebra so that you can see the > PQF queries: > > See > > Bug 15714 : Remove zebra.log from debian scripts and add optional log levels > URL : https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id =15714 > Priority : P5 - low > Urgency : enhancement > Status : RESOLVED > > For setting the log levels > > And http://koha.1045719.n5.nabble.com/ Improving-Zebra-logging-td5861827.html > > For a general discussion of how to use them. > > ... This should give you some idea of what's failing, both in terms of the > dashes in 2.0.59 and the non-functional zebra facets under 2.0.61. > > My general feeling is that 2.0.59 is irredeemably broken by bug 16581, and > we need at least 2.0.60, but I don't have any experience with zebra facets. > > --Barton > > > > On Mon, Aug 22, 2016 at 2:49 PM, Mark Tompsett <mtompset@hotmail.com> wrote: >> >> Greetings, >> >> Similar problem. I hope someone has a better solution than setting it to >> 0. >> >> GPML, >> Mark Tompsett >> >> -----Original Message----- >> From: Philippe Blouin >> Sent: Monday, August 22, 2016 2:40 PM >> To: koha-devel@lists.koha-community.org >> Subject: [Koha-devel] 16.05, zebra and jessie >> >> Hello! >> >> We're trying to find the correction combination. We're new on Jessie, >> so we still have some tweaking needed... >> >> - By default, we get zebra 2.00.59 installed on Jessie through the >> packages. >> - On 16.05, we get some very bad results in the search when the itemtype >> contains an hyphen (-), like 'A-DOC'. >> - So we installed zebra 2.00.62. This fixes the search... >> - But now we do not have facets. >> - So we set <use_zebra_facets>0</use_zebra_facets> >> - And now we have facets. But this feels... wrong? >> >> My dummy question: what is the supposedly correct version of Zebra on >> Jessie ? >> And we're we correct in setting the config to 0 ? >> >> Thanks >> Blou >> _______________________________________________ >> Koha-devel mailing list >> Koha-devel@lists.koha-community.org >> http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel >> website : http://www.koha-community.org/ >> git : http://git.koha-community.org/ >> bugs : http://bugs.koha-community.org/ >> >> _______________________________________________ >> Koha-devel mailing list >> Koha-devel@lists.koha-community.org >> http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel >> website : http://www.koha-community.org/ >> git : http://git.koha-community.org/ >> bugs : http://bugs.koha-community.org/ > > > > _______________________________________________ > Koha-devel mailing list > Koha-devel@lists.koha-community.org > http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > website : http://www.koha-community.org/ > git : http://git.koha-community.org/ > bugs : http://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
--
Tomás Cohen Arazi
Theke Solutions (https://theke.io) ✆ +54 9351 3513384 GPG: B2F3C15F
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Chris Cormack Catalyst IT Ltd. +64 4 803 2238 PO Box 11-053, Manners St, Wellington 6142, New Zealand
I don't doubt that I've missed a ton of words! Between baby and competing projects, I haven't had as much time to keep up with the bleeding edge. I want to be one of those people using and fixing it, although - as you say - everyone has their own priorities. It's the search engine that Koha deserves, but I reckon many of us are busy right now :/. Thanks for the update though, Chris. David Cook Systems Librarian Prosentient Systems 72/330 Wattle St Ultimo, NSW 2007 Australia Office: 02 9212 0899 Direct: 02 8005 0595
-----Original Message----- From: Chris Cormack [mailto:chrisc@catalyst.net.nz] Sent: Wednesday, 31 August 2016 9:28 AM To: David Cook <dcook@prosentient.com.au> Cc: 'Tomas Cohen Arazi' <tomascohen@gmail.com>; 'Barton Chittenden' <barton@bywatersolutions.com>; 'Jonathan Druart' <jonathan.druart@bugs.koha-community.org>; koha-devel@lists.koha- community.org Subject: Re: [Koha-devel] 16.05, zebra and jessie
* David Cook (dcook@prosentient.com.au) wrote:
I suppose Adam at IndexData has been busy with the FOLIO project, so I doubt he has time to work on Zebra these days, even if we did have a
patch available.
Is ElasticSearch usable with Koha at this point? I heard a lot in 2015, but after Robin left I haven’t heard a word other than rumours that the patches had been pushed?
You missed tons of words then :)
Yes, it is in 16.05, marked expiremental, it works, mostly. But it will only get better with more people using and fixing it. The next task is to update the version it works with to Elastic 2. That isn't a huge amount of work, but everyone has their own priorities and a lot of us have to work on what users ask for (not what users need ;))
Chris
Subject: Re: [Koha-devel] 16.05, zebra and jessie
I have seen use_zebra_facets=1 cause no facets rendered when GRS-1 configuration files are kept during upgrades up to where GRS-1 got deprecated (3.20?). Is it the case? What does the About > System information page says about your config?
The slowliness is not in zebra per se, but in the way we retrieve the facets from it (so Koha/Perl side). We retrieve each facet at a time instead of fetching them all in one call. And they come in XML format, so need to be parsed. So, if anyone is willing to improve it, just need to optimize this function (read the TODO):
sub _get_facet_from_result_set {
my $facet_idx = shift;
my $rs = shift;
my $sep = shift;
my $internal_sep = '<*>';
my $facetMaxCount = C4::Context->preference('FacetMaxCount') // 20;
return if ( ! defined $facet_idx || ! defined $rs );
# zebra's facet element, untokenized index
my $facet_element = 'zebra::facet::' . $facet_idx . ':0:' . $facetMaxCount;
# configure zebra results for retrieving the desired facet
$rs->option( elementSetName => $facet_element );
# get the facet record from result set
my $facet = $rs->record( 0 )->raw;
# if the facet has no restuls...
return if !defined $facet;
# TODO: benchmark DOM vs. SAX performance
my $facet_dom = XML::LibXML->load_xml(
string => ($facet)
);
my @terms = $facet_dom->getElementsByTagName('term');
return if ! @terms;
my $facets = {};
foreach my $term ( @terms ) {
my $facet_value = $term->textContent;
$facet_value =~ s/\Q$internal_sep\E/$sep/ if defined $sep;
$facets->{ $facet_value } = $term->getAttribute( 'occur' );
}
return $facets;
}
Another option would be to make _get_facets_from_zebra build the element set containing all facets so they are read in one call (comma-separate all elements). The problem is that Zebra returns zero if one of the elements is empty. Jared proposed to create a ghost record with all facet fields. I didn't manage to make it work. Another option is to patch Zebra. I started that, but abandoned once the ES code
got pushed.
So, if use_zebra_facets=0 is good enough, maybe it should be
recommended.
Problem is it is not a real facet, but the sole extraction of the fields from the first x records.
As I said, it could be good enough anyway.
Regards
El mar., 23 ago. 2016 a las 10:21, Barton Chittenden (< barton@bywatersolutions.com>) escribió:
Zebra tends to be I/O bound -- we've seen it write enormous .zrs files to disk (~16G/query on large libraries). Bug 13665 mentions that searches could be taking upwards of 40 seconds to complete -- I think that we've seen searches time out and return no results at about 1 minute.
Is it possible to tune Zebra's space/time optimizations in any way so that it doesn't write such large files to disk?
On Tue, Aug 23, 2016 at 5:38 AM, Jonathan Druart < jonathan.druart@bugs.koha-community.org> wrote:
See bug 13665 - Retrieve facets from zebra is slow To understand why and when use_zebra_facet=1 is slow
2016-08-22 21:31 GMT+01:00 Barton Chittenden < barton@bywatersolutions.com>: > I haven't run into the issue with the dashes in idzebra-2.0 2.0.59, but I > have run into this, when using ICU-Chains: > > Bug 16581 : ICU tokenization bug in idzebra-2.0 2.0.59-1 > URL : https://bugs.koha- community.org/bugzilla3/show_bug.cgi?id =16581 > Priority : P5 - low > Urgency : enhancement > Status : NEW > > I also know that when use_zebra_facets was first introduced, it was *very* > slow -- I can't find any bugs about that though. It's possible that it got > so slow under idzebra-2.0 2.61 that the searches are timing out. > > It should be possible to set the logging for zebra so that you can see the > PQF queries: > > See > > Bug 15714 : Remove zebra.log from debian scripts and add optional log levels > URL : https://bugs.koha- community.org/bugzilla3/show_bug.cgi?id =15714 > Priority : P5 - low > Urgency : enhancement > Status : RESOLVED > > For setting the log levels > > And http://koha.1045719.n5.nabble.com/ Improving-Zebra-logging-td5861827.html > > For a general discussion of how to use them. > > ... This should give you some idea of what's failing, both in terms of the > dashes in 2.0.59 and the non-functional zebra facets under 2.0.61. > > My general feeling is that 2.0.59 is irredeemably broken by bug 16581, and > we need at least 2.0.60, but I don't have any experience with zebra facets. > > --Barton > > > > On Mon, Aug 22, 2016 at 2:49 PM, Mark Tompsett <mtompset@hotmail.com> wrote: >> >> Greetings, >> >> Similar problem. I hope someone has a better solution than setting it to >> 0. >> >> GPML, >> Mark Tompsett >> >> -----Original Message----- >> From: Philippe Blouin >> Sent: Monday, August 22, 2016 2:40 PM >> To: koha-devel@lists.koha-community.org >> Subject: [Koha-devel] 16.05, zebra and jessie >> >> Hello! >> >> We're trying to find the correction combination. We're new on Jessie, >> so we still have some tweaking needed... >> >> - By default, we get zebra 2.00.59 installed on Jessie through the >> packages. >> - On 16.05, we get some very bad results in the search when the itemtype >> contains an hyphen (-), like 'A-DOC'. >> - So we installed zebra 2.00.62. This fixes the search... >> - But now we do not have facets. >> - So we set <use_zebra_facets>0</use_zebra_facets> >> - And now we have facets. But this feels... wrong? >> >> My dummy question: what is the supposedly correct version of Zebra on >> Jessie ? >> And we're we correct in setting the config to 0 ? >> >> Thanks >> Blou >> _______________________________________________ >> Koha-devel mailing list >> Koha-devel@lists.koha-community.org >> http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha- devel >> website : http://www.koha-community.org/ >> git : http://git.koha-community.org/ >> bugs : http://bugs.koha-community.org/ >> >> _______________________________________________ >> Koha-devel mailing list >> Koha-devel@lists.koha-community.org >> http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha- devel >> website : http://www.koha-community.org/ >> git : http://git.koha-community.org/ >> bugs : http://bugs.koha-community.org/ > > > > _______________________________________________ > Koha-devel mailing list > Koha-devel@lists.koha-community.org > http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > website : http://www.koha-community.org/ > git : http://git.koha-community.org/ > bugs : http://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
--
Tomás Cohen Arazi
Theke Solutions (https://theke.io) ✆ +54 9351 3513384 GPG: B2F3C15F
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Chris Cormack Catalyst IT Ltd. +64 4 803 2238 PO Box 11-053, Manners St, Wellington 6142, New Zealand
2016-08-31 0:19 GMT+01:00 David Cook <dcook@prosentient.com.au>:
Is ElasticSearch usable with Koha at this point? I heard a lot in 2015, but after Robin left I haven’t heard a word other than rumours that the patches had been pushed?
It's pushed and testable. It does not work for new installs (bug 16838) and is broken if OpacSuppression is set (bug 16660). Nobody seems to take care of these bugs (I guess I should). Improvements are provided but don't get lot of attention from signoffers Bug 14567 - Browse interface for OPAC Bug 14899 - Mapping configuration page for Elastic search Cheers, Jonathan
David Cook
Systems Librarian
Prosentient Systems
72/330 Wattle St
Ultimo, NSW 2007
Australia
Office: 02 9212 0899
Direct: 02 8005 0595
From: koha-devel-bounces@lists.koha-community.org [mailto:koha-devel-bounces@lists.koha-community.org] On Behalf Of Tomas Cohen Arazi Sent: Tuesday, 23 August 2016 11:42 PM To: Barton Chittenden <barton@bywatersolutions.com>; Jonathan Druart <jonathan.druart@bugs.koha-community.org> Cc: koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] 16.05, zebra and jessie
I have seen use_zebra_facets=1 cause no facets rendered when GRS-1 configuration files are kept during upgrades up to where GRS-1 got deprecated (3.20?). Is it the case? What does the About > System information page says about your config?
The slowliness is not in zebra per se, but in the way we retrieve the facets from it (so Koha/Perl side). We retrieve each facet at a time instead of fetching them all in one call. And they come in XML format, so need to be parsed. So, if anyone is willing to improve it, just need to optimize this function (read the TODO):
sub _get_facet_from_result_set {
my $facet_idx = shift;
my $rs = shift;
my $sep = shift;
my $internal_sep = '<*>';
my $facetMaxCount = C4::Context->preference('FacetMaxCount') // 20;
return if ( ! defined $facet_idx || ! defined $rs );
# zebra's facet element, untokenized index
my $facet_element = 'zebra::facet::' . $facet_idx . ':0:' . $facetMaxCount;
# configure zebra results for retrieving the desired facet
$rs->option( elementSetName => $facet_element );
# get the facet record from result set
my $facet = $rs->record( 0 )->raw;
# if the facet has no restuls...
return if !defined $facet;
# TODO: benchmark DOM vs. SAX performance
my $facet_dom = XML::LibXML->load_xml(
string => ($facet)
);
my @terms = $facet_dom->getElementsByTagName('term');
return if ! @terms;
my $facets = {};
foreach my $term ( @terms ) {
my $facet_value = $term->textContent;
$facet_value =~ s/\Q$internal_sep\E/$sep/ if defined $sep;
$facets->{ $facet_value } = $term->getAttribute( 'occur' );
}
return $facets;
}
Another option would be to make _get_facets_from_zebra build the element set containing all facets so they are read in one call (comma-separate all elements). The problem is that Zebra returns zero if one of the elements is empty. Jared proposed to create a ghost record with all facet fields. I didn't manage to make it work. Another option is to patch Zebra. I started that, but abandoned once the ES code got pushed.
So, if use_zebra_facets=0 is good enough, maybe it should be recommended. Problem is it is not a real facet, but the sole extraction of the fields from the first x records.
As I said, it could be good enough anyway.
Regards
El mar., 23 ago. 2016 a las 10:21, Barton Chittenden (<barton@bywatersolutions.com>) escribió:
Zebra tends to be I/O bound -- we've seen it write enormous .zrs files to disk (~16G/query on large libraries). Bug 13665 mentions that searches could be taking upwards of 40 seconds to complete -- I think that we've seen searches time out and return no results at about 1 minute.
Is it possible to tune Zebra's space/time optimizations in any way so that it doesn't write such large files to disk?
On Tue, Aug 23, 2016 at 5:38 AM, Jonathan Druart <jonathan.druart@bugs.koha-community.org> wrote:
See bug 13665 - Retrieve facets from zebra is slow To understand why and when use_zebra_facet=1 is slow
2016-08-22 21:31 GMT+01:00 Barton Chittenden <barton@bywatersolutions.com>:
I haven't run into the issue with the dashes in idzebra-2.0 2.0.59, but I have run into this, when using ICU-Chains:
Bug 16581 : ICU tokenization bug in idzebra-2.0 2.0.59-1 URL : https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16581 Priority : P5 - low Urgency : enhancement Status : NEW
I also know that when use_zebra_facets was first introduced, it was *very* slow -- I can't find any bugs about that though. It's possible that it got so slow under idzebra-2.0 2.61 that the searches are timing out.
It should be possible to set the logging for zebra so that you can see the PQF queries:
See
Bug 15714 : Remove zebra.log from debian scripts and add optional log levels URL : https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15714 Priority : P5 - low Urgency : enhancement Status : RESOLVED
For setting the log levels
And http://koha.1045719.n5.nabble.com/Improving-Zebra-logging-td5861827.html
For a general discussion of how to use them.
... This should give you some idea of what's failing, both in terms of the dashes in 2.0.59 and the non-functional zebra facets under 2.0.61.
My general feeling is that 2.0.59 is irredeemably broken by bug 16581, and we need at least 2.0.60, but I don't have any experience with zebra facets.
--Barton
On Mon, Aug 22, 2016 at 2:49 PM, Mark Tompsett <mtompset@hotmail.com> wrote:
Greetings,
Similar problem. I hope someone has a better solution than setting it to 0.
GPML, Mark Tompsett
-----Original Message----- From: Philippe Blouin Sent: Monday, August 22, 2016 2:40 PM To: koha-devel@lists.koha-community.org Subject: [Koha-devel] 16.05, zebra and jessie
Hello!
We're trying to find the correction combination. We're new on Jessie, so we still have some tweaking needed...
- By default, we get zebra 2.00.59 installed on Jessie through the packages. - On 16.05, we get some very bad results in the search when the itemtype contains an hyphen (-), like 'A-DOC'. - So we installed zebra 2.00.62. This fixes the search... - But now we do not have facets. - So we set <use_zebra_facets>0</use_zebra_facets> - And now we have facets. But this feels... wrong?
My dummy question: what is the supposedly correct version of Zebra on Jessie ? And we're we correct in setting the config to 0 ?
Thanks Blou _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
--
Tomás Cohen Arazi
Theke Solutions (https://theke.io) ✆ +54 9351 3513384 GPG: B2F3C15F
We had time and a resources available for a while. We got Bouzid to learn Elastic Search, learn about its integration in Koha, then wanted him to contribute. But been told twice to ... wait. We certainly have no time anymore. We'll be back, but probably not until 2017 unless sponsored. Now, may I suggest something? Documentation. Not user friendly, but at least dev-enlightening. "You" guys put time in there, as with plack and memcache and whatnot, but if you're on the outside, you have no way of getting a head start beside reading some 1200 bugzilla comments in chronological order to know what's the status, how to get it to work on a git, etc... (my ref to plack is not accidental). You can put tens of hours alone on the next bug and then wonder why nobody comes in when you call for testers, or invest a few hours on documenting a quick way for dummasses like me to catch up. *Now*, everyone is constrained by time, and I apologize if the comment above is frustrating to read. I don't want to be rude, and I appreciate all the hours everyone put to improve the product while I don't or can't. Please see that just as my opinion on why different big devs don't get as many helpers as deserved. Philippe Blouin, Responsable du développement informatique Tél. : (888) 604-2627 philippe.blouin@inLibro.com <mailto:philippe.blouin@inLibro.com> inLibro | pour esprit libre | www.inLibro.com <http://www.inLibro.com> On 08/31/2016 03:55 AM, Jonathan Druart wrote:
2016-08-31 0:19 GMT+01:00 David Cook <dcook@prosentient.com.au>:
Is ElasticSearch usable with Koha at this point? I heard a lot in 2015, but after Robin left I haven’t heard a word other than rumours that the patches had been pushed? It's pushed and testable. It does not work for new installs (bug 16838) and is broken if OpacSuppression is set (bug 16660). Nobody seems to take care of these bugs (I guess I should). Improvements are provided but don't get lot of attention from signoffers Bug 14567 - Browse interface for OPAC Bug 14899 - Mapping configuration page for Elastic search
Cheers, Jonathan
David Cook
Systems Librarian
Prosentient Systems
72/330 Wattle St
Ultimo, NSW 2007
Australia
Office: 02 9212 0899
Direct: 02 8005 0595
From: koha-devel-bounces@lists.koha-community.org [mailto:koha-devel-bounces@lists.koha-community.org] On Behalf Of Tomas Cohen Arazi Sent: Tuesday, 23 August 2016 11:42 PM To: Barton Chittenden <barton@bywatersolutions.com>; Jonathan Druart <jonathan.druart@bugs.koha-community.org> Cc: koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] 16.05, zebra and jessie
I have seen use_zebra_facets=1 cause no facets rendered when GRS-1 configuration files are kept during upgrades up to where GRS-1 got deprecated (3.20?). Is it the case? What does the About > System information page says about your config?
The slowliness is not in zebra per se, but in the way we retrieve the facets from it (so Koha/Perl side). We retrieve each facet at a time instead of fetching them all in one call. And they come in XML format, so need to be parsed. So, if anyone is willing to improve it, just need to optimize this function (read the TODO):
sub _get_facet_from_result_set {
my $facet_idx = shift;
my $rs = shift;
my $sep = shift;
my $internal_sep = '<*>';
my $facetMaxCount = C4::Context->preference('FacetMaxCount') // 20;
return if ( ! defined $facet_idx || ! defined $rs );
# zebra's facet element, untokenized index
my $facet_element = 'zebra::facet::' . $facet_idx . ':0:' . $facetMaxCount;
# configure zebra results for retrieving the desired facet
$rs->option( elementSetName => $facet_element );
# get the facet record from result set
my $facet = $rs->record( 0 )->raw;
# if the facet has no restuls...
return if !defined $facet;
# TODO: benchmark DOM vs. SAX performance
my $facet_dom = XML::LibXML->load_xml(
string => ($facet)
);
my @terms = $facet_dom->getElementsByTagName('term');
return if ! @terms;
my $facets = {};
foreach my $term ( @terms ) {
my $facet_value = $term->textContent;
$facet_value =~ s/\Q$internal_sep\E/$sep/ if defined $sep;
$facets->{ $facet_value } = $term->getAttribute( 'occur' );
}
return $facets;
}
Another option would be to make _get_facets_from_zebra build the element set containing all facets so they are read in one call (comma-separate all elements). The problem is that Zebra returns zero if one of the elements is empty. Jared proposed to create a ghost record with all facet fields. I didn't manage to make it work. Another option is to patch Zebra. I started that, but abandoned once the ES code got pushed.
So, if use_zebra_facets=0 is good enough, maybe it should be recommended. Problem is it is not a real facet, but the sole extraction of the fields from the first x records.
As I said, it could be good enough anyway.
Regards
El mar., 23 ago. 2016 a las 10:21, Barton Chittenden (<barton@bywatersolutions.com>) escribió:
Zebra tends to be I/O bound -- we've seen it write enormous .zrs files to disk (~16G/query on large libraries). Bug 13665 mentions that searches could be taking upwards of 40 seconds to complete -- I think that we've seen searches time out and return no results at about 1 minute.
Is it possible to tune Zebra's space/time optimizations in any way so that it doesn't write such large files to disk?
On Tue, Aug 23, 2016 at 5:38 AM, Jonathan Druart <jonathan.druart@bugs.koha-community.org> wrote:
See bug 13665 - Retrieve facets from zebra is slow To understand why and when use_zebra_facet=1 is slow
2016-08-22 21:31 GMT+01:00 Barton Chittenden <barton@bywatersolutions.com>:
I haven't run into the issue with the dashes in idzebra-2.0 2.0.59, but I have run into this, when using ICU-Chains:
Bug 16581 : ICU tokenization bug in idzebra-2.0 2.0.59-1 URL : https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16581 Priority : P5 - low Urgency : enhancement Status : NEW
I also know that when use_zebra_facets was first introduced, it was *very* slow -- I can't find any bugs about that though. It's possible that it got so slow under idzebra-2.0 2.61 that the searches are timing out.
It should be possible to set the logging for zebra so that you can see the PQF queries:
See
Bug 15714 : Remove zebra.log from debian scripts and add optional log levels URL : https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15714 Priority : P5 - low Urgency : enhancement Status : RESOLVED
For setting the log levels
And http://koha.1045719.n5.nabble.com/Improving-Zebra-logging-td5861827.html
For a general discussion of how to use them.
... This should give you some idea of what's failing, both in terms of the dashes in 2.0.59 and the non-functional zebra facets under 2.0.61.
My general feeling is that 2.0.59 is irredeemably broken by bug 16581, and we need at least 2.0.60, but I don't have any experience with zebra facets.
--Barton
On Mon, Aug 22, 2016 at 2:49 PM, Mark Tompsett <mtompset@hotmail.com> wrote:
Greetings,
Similar problem. I hope someone has a better solution than setting it to 0.
GPML, Mark Tompsett
-----Original Message----- From: Philippe Blouin Sent: Monday, August 22, 2016 2:40 PM To: koha-devel@lists.koha-community.org Subject: [Koha-devel] 16.05, zebra and jessie
Hello!
We're trying to find the correction combination. We're new on Jessie, so we still have some tweaking needed...
- By default, we get zebra 2.00.59 installed on Jessie through the packages. - On 16.05, we get some very bad results in the search when the itemtype contains an hyphen (-), like 'A-DOC'. - So we installed zebra 2.00.62. This fixes the search... - But now we do not have facets. - So we set <use_zebra_facets>0</use_zebra_facets> - And now we have facets. But this feels... wrong?
My dummy question: what is the supposedly correct version of Zebra on Jessie ? And we're we correct in setting the config to 0 ?
Thanks Blou _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
--
Tomás Cohen Arazi
Theke Solutions (https://theke.io) ✆ +54 9351 3513384 GPG: B2F3C15F
Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
El mié., 31 ago. 2016 a las 9:17, Philippe Blouin (< philippe.blouin@inlibro.com>) escribió:
We had time and a resources available for a while. We got Bouzid to learn Elastic Search, learn about its integration in Koha, then wanted him to contribute. But been told twice to ... wait.
I don't think that's the common behaviour in the dev group. Everyone does whatever they want and tries to get attention on what they did, synergy. Sometimes when one is doing some dev and others pretend to do it again, you tell'em to hold on, so there's no overlap. But anyone willing to improve the project is usually encouraged. Maybe a communication issue.
We certainly have no time anymore. We'll be back, but probably not until 2017 unless sponsored.
Well, at least you will be able soon :-D
Now, may I suggest something? Documentation. Not user friendly, but at least dev-enlightening.
"You" guys put time in there, as with plack and memcache and whatnot, but if you're on the outside, you have no way of getting a head start beside reading some 1200 bugzilla comments in chronological order to know what's the status, how to get it to work on a git, etc... (my ref to plack is not accidental).
As a non-Plack-expert (but got involved on integrating it and fixing stuff) I agree the conversation about Plack was difficult. Some problems are really tricky and difficult to explain. You can put tens of hours alone on the next bug and then wonder why nobody
comes in when you call for testers, or invest a few hours on documenting a quick way for dummasses like me to catch up.
The best option is asking for help to get involved. I tend to translate silence<->lack of interest. But maybe that's just my problem or a different-culture problem. I remember I sent several emails when trying to integrate Plack into the packages and didn't get much attention. Maybe the question about the design decisions we need to make about permission checking gets more attention, because it is more a theoretical question. Regards -- Tomás Cohen Arazi Theke Solutions (https://theke.io <http://theke.io/>) ✆ +54 9351 3513384 GPG: B2F3C15F
On Wed, Aug 31, 2016 at 8:17 AM, Philippe Blouin < philippe.blouin@inlibro.com> wrote:
Now, may I suggest something? Documentation. Not user friendly, but at least dev-enlightening.
"You" guys put time in there, as with plack and memcache and whatnot, but if you're on the outside, you have no way of getting a head start beside reading some 1200 bugzilla comments in chronological order to know what's the status, how to get it to work on a git, etc... (my ref to plack is not accidental).
You can put tens of hours alone on the next bug and then wonder why nobody comes in when you call for testers, or invest a few hours on documenting a quick way for dummasses like me to catch up.
*Now*, everyone is constrained by time, and I apologize if the comment above is frustrating to read. I don't want to be rude, and I appreciate all the hours everyone put to improve the product while I don't or can't. Please see that just as my opinion on why different big devs don't get as many helpers as deserved.
I think this is actually a very cogent observation, and deserves its own thread. I've definitely been in the situation where I've wanted a better view of large developments, but was overwhelmed by the comment threads in bugzilla. I've got some ideas, but I don't want to hijack the discussion of 16.05, zebra and jessie.
Philippe, If you need help for testing something, please ask on the ML or #koha, same if you need documentation to setup something. One of us might have the answer. Going to a same direction at a same time is certainly one of the biggest problem we have in the Koha community. To let an overview of what we (the devs) are doing currently, my advice is to read the "what's on" email I have been emailing for a few months. The goal is to let know people who wants to help where they can help. In the medium term, I hope it will help to concentrate efforts on a same problematic and to help us focus all together on the same things. Cheers, Jonathan 2016-08-31 13:17 GMT+01:00 Philippe Blouin <philippe.blouin@inlibro.com>:
We had time and a resources available for a while. We got Bouzid to learn Elastic Search, learn about its integration in Koha, then wanted him to contribute. But been told twice to ... wait.
We certainly have no time anymore. We'll be back, but probably not until 2017 unless sponsored.
Now, may I suggest something? Documentation. Not user friendly, but at least dev-enlightening.
"You" guys put time in there, as with plack and memcache and whatnot, but if you're on the outside, you have no way of getting a head start beside reading some 1200 bugzilla comments in chronological order to know what's the status, how to get it to work on a git, etc... (my ref to plack is not accidental).
You can put tens of hours alone on the next bug and then wonder why nobody comes in when you call for testers, or invest a few hours on documenting a quick way for dummasses like me to catch up.
Now, everyone is constrained by time, and I apologize if the comment above is frustrating to read. I don't want to be rude, and I appreciate all the hours everyone put to improve the product while I don't or can't. Please see that just as my opinion on why different big devs don't get as many helpers as deserved.
Philippe Blouin, Responsable du développement informatique
Tél. : (888) 604-2627 philippe.blouin@inLibro.com
inLibro | pour esprit libre | www.inLibro.com On 08/31/2016 03:55 AM, Jonathan Druart wrote:
2016-08-31 0:19 GMT+01:00 David Cook <dcook@prosentient.com.au>:
Is ElasticSearch usable with Koha at this point? I heard a lot in 2015, but after Robin left I haven’t heard a word other than rumours that the patches had been pushed?
It's pushed and testable. It does not work for new installs (bug 16838) and is broken if OpacSuppression is set (bug 16660). Nobody seems to take care of these bugs (I guess I should). Improvements are provided but don't get lot of attention from signoffers Bug 14567 - Browse interface for OPAC Bug 14899 - Mapping configuration page for Elastic search
Cheers, Jonathan
David Cook
Systems Librarian
Prosentient Systems
72/330 Wattle St
Ultimo, NSW 2007
Australia
Office: 02 9212 0899
Direct: 02 8005 0595
From: koha-devel-bounces@lists.koha-community.org [mailto:koha-devel-bounces@lists.koha-community.org] On Behalf Of Tomas Cohen Arazi Sent: Tuesday, 23 August 2016 11:42 PM To: Barton Chittenden <barton@bywatersolutions.com>; Jonathan Druart <jonathan.druart@bugs.koha-community.org> Cc: koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] 16.05, zebra and jessie
I have seen use_zebra_facets=1 cause no facets rendered when GRS-1 configuration files are kept during upgrades up to where GRS-1 got deprecated (3.20?). Is it the case? What does the About > System information page says about your config?
The slowliness is not in zebra per se, but in the way we retrieve the facets from it (so Koha/Perl side). We retrieve each facet at a time instead of fetching them all in one call. And they come in XML format, so need to be parsed. So, if anyone is willing to improve it, just need to optimize this function (read the TODO):
sub _get_facet_from_result_set {
my $facet_idx = shift;
my $rs = shift;
my $sep = shift;
my $internal_sep = '<*>';
my $facetMaxCount = C4::Context->preference('FacetMaxCount') // 20;
return if ( ! defined $facet_idx || ! defined $rs );
# zebra's facet element, untokenized index
my $facet_element = 'zebra::facet::' . $facet_idx . ':0:' . $facetMaxCount;
# configure zebra results for retrieving the desired facet
$rs->option( elementSetName => $facet_element );
# get the facet record from result set
my $facet = $rs->record( 0 )->raw;
# if the facet has no restuls...
return if !defined $facet;
# TODO: benchmark DOM vs. SAX performance
my $facet_dom = XML::LibXML->load_xml(
string => ($facet)
);
my @terms = $facet_dom->getElementsByTagName('term');
return if ! @terms;
my $facets = {};
foreach my $term ( @terms ) {
my $facet_value = $term->textContent;
$facet_value =~ s/\Q$internal_sep\E/$sep/ if defined $sep;
$facets->{ $facet_value } = $term->getAttribute( 'occur' );
}
return $facets;
}
Another option would be to make _get_facets_from_zebra build the element set containing all facets so they are read in one call (comma-separate all elements). The problem is that Zebra returns zero if one of the elements is empty. Jared proposed to create a ghost record with all facet fields. I didn't manage to make it work. Another option is to patch Zebra. I started that, but abandoned once the ES code got pushed.
So, if use_zebra_facets=0 is good enough, maybe it should be recommended. Problem is it is not a real facet, but the sole extraction of the fields from the first x records.
As I said, it could be good enough anyway.
Regards
El mar., 23 ago. 2016 a las 10:21, Barton Chittenden (<barton@bywatersolutions.com>) escribió:
Zebra tends to be I/O bound -- we've seen it write enormous .zrs files to disk (~16G/query on large libraries). Bug 13665 mentions that searches could be taking upwards of 40 seconds to complete -- I think that we've seen searches time out and return no results at about 1 minute.
Is it possible to tune Zebra's space/time optimizations in any way so that it doesn't write such large files to disk?
On Tue, Aug 23, 2016 at 5:38 AM, Jonathan Druart <jonathan.druart@bugs.koha-community.org> wrote:
See bug 13665 - Retrieve facets from zebra is slow To understand why and when use_zebra_facet=1 is slow
2016-08-22 21:31 GMT+01:00 Barton Chittenden <barton@bywatersolutions.com>:
I haven't run into the issue with the dashes in idzebra-2.0 2.0.59, but I have run into this, when using ICU-Chains:
Bug 16581 : ICU tokenization bug in idzebra-2.0 2.0.59-1 URL : https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16581 Priority : P5 - low Urgency : enhancement Status : NEW
I also know that when use_zebra_facets was first introduced, it was *very* slow -- I can't find any bugs about that though. It's possible that it got so slow under idzebra-2.0 2.61 that the searches are timing out.
It should be possible to set the logging for zebra so that you can see the PQF queries:
See
Bug 15714 : Remove zebra.log from debian scripts and add optional log levels URL : https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15714 Priority : P5 - low Urgency : enhancement Status : RESOLVED
For setting the log levels
And http://koha.1045719.n5.nabble.com/Improving-Zebra-logging-td5861827.html
For a general discussion of how to use them.
... This should give you some idea of what's failing, both in terms of the dashes in 2.0.59 and the non-functional zebra facets under 2.0.61.
My general feeling is that 2.0.59 is irredeemably broken by bug 16581, and we need at least 2.0.60, but I don't have any experience with zebra facets.
--Barton
On Mon, Aug 22, 2016 at 2:49 PM, Mark Tompsett <mtompset@hotmail.com> wrote:
Greetings,
Similar problem. I hope someone has a better solution than setting it to 0.
GPML, Mark Tompsett
-----Original Message----- From: Philippe Blouin Sent: Monday, August 22, 2016 2:40 PM To: koha-devel@lists.koha-community.org Subject: [Koha-devel] 16.05, zebra and jessie
Hello!
We're trying to find the correction combination. We're new on Jessie, so we still have some tweaking needed...
- By default, we get zebra 2.00.59 installed on Jessie through the packages. - On 16.05, we get some very bad results in the search when the itemtype contains an hyphen (-), like 'A-DOC'. - So we installed zebra 2.00.62. This fixes the search... - But now we do not have facets. - So we set <use_zebra_facets>0</use_zebra_facets> - And now we have facets. But this feels... wrong?
My dummy question: what is the supposedly correct version of Zebra on Jessie ? And we're we correct in setting the config to 0 ?
Thanks Blou _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
--
Tomás Cohen Arazi
Theke Solutions (https://theke.io) ✆ +54 9351 3513384 GPG: B2F3C15F
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
participants (10)
-
Barton Chittenden -
Chris Cormack -
David Cook -
Jonathan Druart -
Katrin Fischer -
Kyle Hall -
Marcel de Rooy -
Mark Tompsett -
Philippe Blouin -
Tomas Cohen Arazi