First some ranting about Zebra and then a question about ElasticSearch. Using Zebra there is a Global system pref. called maxRecordsForFacets. By default the number of records used to build the facet list is the first 20 results shown. You also get a warning in the manual to increase the number since it will impact respons time. Building a facet list out of 20 records may be enough for some people but if you think twice about it you realise that it is quite misleading, especially since the user does not know that the facet list is built on a very limited selection of records. Having the sort set on Author and searching for realivity theory, the facet list may not show Einstein if your library is specialised in physics and have a fairly large collection. I have high hopes for ElasticSearch to work with facets in a better way. The best would be if all the index could be used for producing e.g. the author facet and that it would have minimal impact on system performance. I understand that there is discussion about making facets configurable (bug 18235) but I cannot see any traces of the improvements I would like to see, but I may be wrong since I am quite new working with Koha. Regards Claes Eriksson, VTI, Sweden
Hi Claes, I think the maxRecordsForFacets system preference might be a bit legacy at this point. It was first used when we were building facets manually from retrieved records from Zebra (which was an awful awful no good process). However, I think most installations using Zebra these days use Zebra's native faceting which uses the entire result set. At least that's how things were a year or two ago. It may have changed since then. I'd have to double-check, and I have a lot on at the moment, so I'll leave that for yourself or someone else to check. In any case, we're all aware of the limitations around that system preference, which is why we made the move to Zebra's native faceting. (One easy thing to check is in your koha-conf.xml file to see if "use_zebra_facets" is set to 1. If so, then I think "maxRecordsForFacets" shouldn't have any effect. I'm not very familiar with the ElasticSearch efforts, but I think it started out as a like-for-like drop-in replacement. Making facets configurable would be great, but I'm not sure anyone has this as a priority at the moment. I think for now the main goal of ElasticSearch may be stability? David Cook Systems Librarian Prosentient Systems 72/330 Wattle St Ultimo, NSW 2007 Australia Office: 02 9212 0899 Direct: 02 8005 0595 From: koha-devel-bounces@lists.koha-community.org [mailto:koha-devel-bounces@lists.koha-community.org] On Behalf Of Claes Eriksson Sent: Monday, 19 February 2018 11:01 PM To: koha-devel@lists.koha-community.org Subject: [Koha-devel] Elasticsearch and facets First some ranting about Zebra and then a question about ElasticSearch. Using Zebra there is a Global system pref. called maxRecordsForFacets. By default the number of records used to build the facet list is the first 20 results shown. You also get a warning in the manual to increase the number since it will impact respons time. Building a facet list out of 20 records may be enough for some people but if you think twice about it you realise that it is quite misleading, especially since the user does not know that the facet list is built on a very limited selection of records. Having the sort set on Author and searching for realivity theory, the facet list may not show Einstein if your library is specialised in physics and have a fairly large collection. I have high hopes for ElasticSearch to work with facets in a better way. The best would be if all the index could be used for producing e.g. the author facet and that it would have minimal impact on system performance. I understand that there is discussion about making facets configurable (bug 18235) but I cannot see any traces of the improvements I would like to see, but I may be wrong since I am quite new working with Koha. Regards Claes Eriksson, VTI, Sweden
Facets are configurable, and goal is feature parity. El lun., 19 de feb. de 2018 8:59 p. m., David Cook <dcook@prosentient.com.au> escribió:
Hi Claes,
I think the maxRecordsForFacets system preference might be a bit legacy at this point. It was first used when we were building facets manually from retrieved records from Zebra (which was an awful awful no good process). However, I think most installations using Zebra these days use Zebra’s native faceting which uses the entire result set. At least that’s how things were a year or two ago. It may have changed since then. I’d have to double-check, and I have a lot on at the moment, so I’ll leave that for yourself or someone else to check. In any case, we’re all aware of the limitations around that system preference, which is why we made the move to Zebra’s native faceting. (One easy thing to check is in your koha-conf.xml file to see if “use_zebra_facets” is set to 1. If so, then I think “maxRecordsForFacets” shouldn’t have any effect.
I’m not very familiar with the ElasticSearch efforts, but I think it started out as a like-for-like drop-in replacement. Making facets configurable would be great, but I’m not sure anyone has this as a priority at the moment. I think for now the main goal of ElasticSearch may be stability?
David Cook
Systems Librarian
Prosentient Systems
72/330 Wattle St
Ultimo, NSW 2007
Australia
Office: 02 9212 0899
Direct: 02 8005 0595
*From:* koha-devel-bounces@lists.koha-community.org [mailto: koha-devel-bounces@lists.koha-community.org] *On Behalf Of *Claes Eriksson *Sent:* Monday, 19 February 2018 11:01 PM *To:* koha-devel@lists.koha-community.org *Subject:* [Koha-devel] Elasticsearch and facets
First some ranting about Zebra and then a question about ElasticSearch.
Using Zebra there is a Global system pref. called maxRecordsForFacets. By default the number of records used to build the facet list is the first 20 results shown. You also get a warning in the manual to increase the number since it will impact respons time. Building a facet list out of 20 records may be enough for some people but if you think twice about it you realise that it is quite misleading, especially since the user does not know that the facet list is built on a very limited selection of records. Having the sort set on Author and searching for realivity theory, the facet list may not show Einstein if your library is specialised in physics and have a fairly large collection.
I have high hopes for ElasticSearch to work with facets in a better way. The best would be if all the index could be used for producing e.g. the author facet and that it would have minimal impact on system performance. I understand that there is discussion about making facets configurable (bug 18235) but I cannot see any traces of the improvements I would like to see, but I may be wrong since I am quite new working with Koha.
Regards
Claes Eriksson, VTI, Sweden
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Tomás Cohen Arazi Theke Solutions (https://theke.io <http://theke.io/>) ✆ +54 9351 3513384 GPG: B2F3C15F
1. To begin with: “use_zebra_facets” was set to 1 in koha-conf.xml file, so that was not the problem. 2. Regarding Zebra’s native faceting I find some traces in an old RCF saying that it would be nice to replace "homegrown and inefficient routines for faceting" with Zebra’s native faceting. https://wiki.koha-community.org/wiki/C_%26_P_Search_Rewrite_RFC#.28e.29_Impl.... 3. I have also checked some libraries around the world with Koha set up by some of the larger consultants and they all handle facets in this way. I may ad to my ranting that it becomes a serious problem when it comes to subject facets. Narrowing down a search using subject facets that does only reflect the first 20 results is not helping a student or researcher. BTW, we tested setting default to 50 first results and response time increased with 5-8 seconds = not acceptable. 4. As I understand facets are hardcoded in Zebra and not at all configurable. If I am wrong I would be most grateful if you could hint me in the right direction. In ElasticSearch it is under discussion and not solved according to bug 18235? The best Claes ------------------------------------ Claes Eriksson Information specialist Department of Communication, Marketing and Library Swedish National Road and Transport Research Institute (VTI) VTI, SE-581 95 Linkoeping, Sweden Tel: +46-13-20 40 00 Direct: +46-13-20 41 99 E-mail: claes.eriksson@vti.se<mailto:claes.eriksson@vti.se> ________________________________ Från: Tomas Cohen Arazi <tomascohen@gmail.com> Skickat: den 20 februari 2018 01:13:18 Till: David Cook Kopia: Claes Eriksson; koha-devel@lists.koha-community.org Ämne: Re: [Koha-devel] Elasticsearch and facets Facets are configurable, and goal is feature parity. El lun., 19 de feb. de 2018 8:59 p. m., David Cook <dcook@prosentient.com.au<mailto:dcook@prosentient.com.au>> escribió: Hi Claes, I think the maxRecordsForFacets system preference might be a bit legacy at this point. It was first used when we were building facets manually from retrieved records from Zebra (which was an awful awful no good process). However, I think most installations using Zebra these days use Zebra’s native faceting which uses the entire result set. At least that’s how things were a year or two ago. It may have changed since then. I’d have to double-check, and I have a lot on at the moment, so I’ll leave that for yourself or someone else to check. In any case, we’re all aware of the limitations around that system preference, which is why we made the move to Zebra’s native faceting. (One easy thing to check is in your koha-conf.xml file to see if “use_zebra_facets” is set to 1. If so, then I think “maxRecordsForFacets” shouldn’t have any effect. I’m not very familiar with the ElasticSearch efforts, but I think it started out as a like-for-like drop-in replacement. Making facets configurable would be great, but I’m not sure anyone has this as a priority at the moment. I think for now the main goal of ElasticSearch may be stability? David Cook Systems Librarian Prosentient Systems 72/330 Wattle St Ultimo, NSW 2007 Australia Office: 02 9212 0899 Direct: 02 8005 0595 From: koha-devel-bounces@lists.koha-community.org<mailto:koha-devel-bounces@lists.koha-community.org> [mailto:koha-devel-bounces@lists.koha-community.org<mailto:koha-devel-bounces@lists.koha-community.org>] On Behalf Of Claes Eriksson Sent: Monday, 19 February 2018 11:01 PM To: koha-devel@lists.koha-community.org<mailto:koha-devel@lists.koha-community.org> Subject: [Koha-devel] Elasticsearch and facets First some ranting about Zebra and then a question about ElasticSearch. Using Zebra there is a Global system pref. called maxRecordsForFacets. By default the number of records used to build the facet list is the first 20 results shown. You also get a warning in the manual to increase the number since it will impact respons time. Building a facet list out of 20 records may be enough for some people but if you think twice about it you realise that it is quite misleading, especially since the user does not know that the facet list is built on a very limited selection of records. Having the sort set on Author and searching for realivity theory, the facet list may not show Einstein if your library is specialised in physics and have a fairly large collection. I have high hopes for ElasticSearch to work with facets in a better way. The best would be if all the index could be used for producing e.g. the author facet and that it would have minimal impact on system performance. I understand that there is discussion about making facets configurable (bug 18235) but I cannot see any traces of the improvements I would like to see, but I may be wrong since I am quite new working with Koha. Regards Claes Eriksson, VTI, Sweden _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org<mailto:Koha-devel@lists.koha-community.org> http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/ -- Tomás Cohen Arazi Theke Solutions (https://theke.io<http://theke.io/>) ✆ +54 9351 3513384 GPG: B2F3C15F
Regarding Zebra’s native faceting I find some traces in an old RCF saying
El mar., 20 feb. 2018 a las 10:23, Claes Eriksson (<claes.eriksson@vti.se>) escribió: that it would
be nice to replace "homegrown and inefficient routines for faceting" with Zebra’s native faceting.
Zebra facets usage was implemented here: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11232
I have also checked some libraries around the world with Koha set up by some of the larger consultants and they all handle facets in this way. I may ad to my ranting that it becomes a serious problem when it comes to subject facets. Narrowing down a search using subject facets that does only reflect the first 20 results is not helping a student or researcher. BTW, we tested setting default to 50 first results and response time increased with 5-8 seconds = not acceptable.
If you are using Zebra's native facets implementation (use_zebra_facets) on a recent Koha version (i.e. using DOM indexing) then your system is not using maxRecordsForFacets, but this configuration: https://gitlab.com/koha-community/Koha/blob/master/etc/zebradb/zebra-biblios... As you can see, it is pulling the facets from the first 1000 records in the result set. You could definitely try lowering it down to something reasonable (in your terms).
As I understand facets are hardcoded in Zebra and not at all configurable. If I am wrong I would > be most grateful if you could hint me in the right direction. In ElasticSearch it is under discussion and not solved according to bug 18235?
Sorry for the mistake, I had my eyes on that bug a while ago and was sure there was lots of interest in the feature, so assumed it was already in master! Good luck. -- Tomás Cohen Arazi Theke Solutions (https://theke.io <http://theke.io/>) ✆ +54 9351 3513384 GPG: B2F3C15F
Hello Claes, About Zebra first: maxRecordsForFacets is only used if facets are not built from Zebra (use_zebra_facets=0). If facets are retrieved from Zebra you may face performance issue depending on the number of records you have in your catalogue (see bug 13665). You can then play with the parameter "facetNumRecs" in your zebra config file (default is 1000). Regarding Elastic: Facets are not configurable yet, but will be with the patches on bug 18235. It is a priority for me and that is why I proposed a patch one year ago. You can easily test it and provide us feedback. Regards, Jonathan On Tue, 20 Feb 2018 at 10:23 Claes Eriksson <claes.eriksson@vti.se> wrote:
1. To begin with: “use_zebra_facets” was set to 1 in koha-conf.xml file, so that was not the problem. 2. Regarding Zebra’s native faceting I find some traces in an old RCF saying that it would be nice to replace "homegrown and inefficient routines for faceting" with Zebra’s native faceting. https://wiki.koha-community.org/wiki/C_%26_P_Search_Rewrite_RFC#.28e.29_Impl... . 3. I have also checked some libraries around the world with Koha set up by some of the larger consultants and they all handle facets in this way. I may ad to my ranting that it becomes a serious problem when it comes to subject facets. Narrowing down a search using subject facets that does only reflect the first 20 results is not helping a student or researcher. BTW, we tested setting default to 50 first results and response time increased with 5-8 seconds = not acceptable. 4. As I understand facets are hardcoded in Zebra and not at all configurable. If I am wrong I would be most grateful if you could hint me in the right direction. In ElasticSearch it is under discussion and not solved according to bug 18235?
The best
Claes
------------------------------------ *Claes Eriksson* Information specialist
Department of Communication, Marketing and Library
*Swedish National Road and Transport Research Institute* (VTI) VTI, SE-581 95 Linkoeping, Sweden Tel: +46-13-20 40 00 <+46%2013%2020%2040%2000> Direct: +46-13-20 41 99 <+46%2013%2020%2041%2099> E-mail: claes.eriksson@vti.se ------------------------------
*Från:* Tomas Cohen Arazi <tomascohen@gmail.com> *Skickat:* den 20 februari 2018 01:13:18 *Till:* David Cook *Kopia:* Claes Eriksson; koha-devel@lists.koha-community.org *Ämne:* Re: [Koha-devel] Elasticsearch and facets
Facets are configurable, and goal is feature parity.
El lun., 19 de feb. de 2018 8:59 p. m., David Cook < dcook@prosentient.com.au> escribió:
Hi Claes,
I think the maxRecordsForFacets system preference might be a bit legacy at this point. It was first used when we were building facets manually from retrieved records from Zebra (which was an awful awful no good process). However, I think most installations using Zebra these days use Zebra’s native faceting which uses the entire result set. At least that’s how things were a year or two ago. It may have changed since then. I’d have to double-check, and I have a lot on at the moment, so I’ll leave that for yourself or someone else to check. In any case, we’re all aware of the limitations around that system preference, which is why we made the move to Zebra’s native faceting. (One easy thing to check is in your koha-conf.xml file to see if “use_zebra_facets” is set to 1. If so, then I think “maxRecordsForFacets” shouldn’t have any effect.
I’m not very familiar with the ElasticSearch efforts, but I think it started out as a like-for-like drop-in replacement. Making facets configurable would be great, but I’m not sure anyone has this as a priority at the moment. I think for now the main goal of ElasticSearch may be stability?
David Cook
Systems Librarian
Prosentient Systems
72/330 Wattle St
Ultimo, NSW 2007
Australia
Office: 02 9212 0899 <02%2092%2012%2008%2099>
Direct: 02 8005 0595 <02%2080%2005%2005%2095>
*From:* koha-devel-bounces@lists.koha-community.org [mailto: koha-devel-bounces@lists.koha-community.org] *On Behalf Of *Claes Eriksson *Sent:* Monday, 19 February 2018 11:01 PM *To:* koha-devel@lists.koha-community.org *Subject:* [Koha-devel] Elasticsearch and facets
First some ranting about Zebra and then a question about ElasticSearch.
Using Zebra there is a Global system pref. called maxRecordsForFacets. By default the number of records used to build the facet list is the first 20 results shown. You also get a warning in the manual to increase the number since it will impact respons time. Building a facet list out of 20 records may be enough for some people but if you think twice about it you realise that it is quite misleading, especially since the user does not know that the facet list is built on a very limited selection of records. Having the sort set on Author and searching for realivity theory, the facet list may not show Einstein if your library is specialised in physics and have a fairly large collection.
I have high hopes for ElasticSearch to work with facets in a better way. The best would be if all the index could be used for producing e.g. the author facet and that it would have minimal impact on system performance. I understand that there is discussion about making facets configurable (bug 18235) but I cannot see any traces of the improvements I would like to see, but I may be wrong since I am quite new working with Koha.
Regards
Claes Eriksson, VTI, Sweden
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
--
Tomás Cohen Arazi
Theke Solutions (https://theke.io <http://theke.io/>) ✆ +54 9351 3513384 <+54%209%20351%20351-3384> GPG: B2F3C15F _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
What's the DB size BTW? I have a couple sites with 200k biblios and everything seems below 2sec. El lun., 19 feb. 2018 a las 9:01, Claes Eriksson (<claes.eriksson@vti.se>) escribió:
First some ranting about Zebra and then a question about ElasticSearch.
Using Zebra there is a Global system pref. called maxRecordsForFacets. By default the number of records used to build the facet list is the first 20 results shown. You also get a warning in the manual to increase the number since it will impact respons time. Building a facet list out of 20 records may be enough for some people but if you think twice about it you realise that it is quite misleading, especially since the user does not know that the facet list is built on a very limited selection of records. Having the sort set on Author and searching for realivity theory, the facet list may not show Einstein if your library is specialised in physics and have a fairly large collection.
I have high hopes for ElasticSearch to work with facets in a better way. The best would be if all the index could be used for producing e.g. the author facet and that it would have minimal impact on system performance. I understand that there is discussion about making facets configurable (bug 18235) but I cannot see any traces of the improvements I would like to see, but I may be wrong since I am quite new working with Koha.
Regards Claes Eriksson, VTI, Sweden
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Tomás Cohen Arazi Theke Solutions (https://theke.io <http://theke.io/>) ✆ +54 9351 3513384 GPG: B2F3C15F
participants (4)
-
Claes Eriksson -
David Cook -
Jonathan Druart -
Tomas Cohen Arazi