Hello, I have tested the QueryStemming system preference on Koha 3.14 (my local installation) and I'm wondering, does zebra just right truncate the words or is there an algorithm to find the stems? I use ICU and I have enabled "QueryWeightFields". I don't have automatic truncation or fuzzy search on. I use these words for my tests: * ski, skiing, skills * fish, fished, fishing, fisher, fishxsdfe Each time, with QueryStemming on, skills and fishxsdfe come out in the search results. Is it what I should expect? "Skills", maybe but "fishxsdfe"? Do you know how it works? or have a good example that would help me to understand? Thanks! -- François Charbonnier, Bibl. prof. / Chef de produits Tél. : (888) 604-2627 francois.charbonnier@inLibro.com <mailto:francois.charbonnier@inLibro.com> inLibro | pour esprit libre | www.inLibro.com <http://www.inLibro.com>
Hi Francois Writing from a tablet so I will be brief. If you’re not using queryparser, there is a Perl module that does the stemming I believe. Lingua::Snowball or something like that. It would be easy to add to the queryparser too but haven’t gotten to it yet. Oh… now that I think about it… I think we mangle keywords before stemming them. I can’t remember why but there was some reason for it. Sorry this message is so incoherent. Will investigate and type a better response when I have a proper keyboard and not just thumbs on a small screen. David From: koha-devel-bounces@lists.koha-community.org [mailto:koha-devel-bounces@lists.koha-community.org] On Behalf Of Francois Charbonnier Sent: Wednesday, 27 August 2014 2:09 AM To: koha-devel@lists.koha-community.org Subject: [Koha-devel] Stemming and zebra Hello, I have tested the QueryStemming system preference on Koha 3.14 (my local installation) and I'm wondering, does zebra just right truncate the words or is there an algorithm to find the stems? I use ICU and I have enabled "QueryWeightFields". I don't have automatic truncation or fuzzy search on. I use these words for my tests: * ski, skiing, skills * fish, fished, fishing, fisher, fishxsdfe Each time, with QueryStemming on, skills and fishxsdfe come out in the search results. Is it what I should expect? "Skills", maybe but "fishxsdfe"? Do you know how it works? or have a good example that would help me to understand? Thanks! -- François Charbonnier, Bibl. prof. / Chef de produits Tél. : (888) 604-2627 <mailto:francois.charbonnier@inLibro.com> francois.charbonnier@inLibro.com inLibro | pour esprit libre | <http://www.inLibro.com> www.inLibro.com
Hi Francois: I wrote an email earlier on my tablet, but not 100% sure if it got sent. In any case, I’m writing again now! You’ll want to look at C4::Search::_build_stemmed_operand(). Zebra doesn’t actually do any stemming itself. If you read through the Zebra docs (if you’re masochistic), you’ll notice that they say explicitly that Zebra doesn’t do any stemming, but that you can do stemming (using a stemmer like Snowball) while building a query. That’s exactly what we do in Koha. The Perl module that does the stemming is Lingua::Stem::Snowball. However, you might notice that your query’s operands aren’t always stemmed properly. I haven’t looked in a while, but I think it’s because we don’t build our queries very well at all (when not using QueryParser). If you want to understand why you’re getting “skills” and “fishxsdfe” in your results, I would suggest running some tests ( using “Data::Dumper” and “warn” ) so that you can see your query as it is built. I have a lot of work I want to do on C4::Search::buildQuery, but just don’t have the time :/. Unfortunately, at the moment, there is no stemming when using the QueryParser. However, fortunately, using Lingua::Stem::Snowball with QueryParser would be really really easy. I think that I’ve written a note on how to do that somewhere on Bugzilla or maybe on Trello… I hope that helps! Feel free to send me an email or shout at me on IRC if you want any clarification. I know I probably didn’t make it any clearer but hopefully this might help you on your path to understanding. David Cook Systems Librarian Prosentient Systems 72/330 Wattle St, Ultimo, NSW 2007 From: koha-devel-bounces@lists.koha-community.org [mailto:koha-devel-bounces@lists.koha-community.org] On Behalf Of Francois Charbonnier Sent: Wednesday, 27 August 2014 2:09 AM To: koha-devel@lists.koha-community.org Subject: [Koha-devel] Stemming and zebra Hello, I have tested the QueryStemming system preference on Koha 3.14 (my local installation) and I'm wondering, does zebra just right truncate the words or is there an algorithm to find the stems? I use ICU and I have enabled "QueryWeightFields". I don't have automatic truncation or fuzzy search on. I use these words for my tests: * ski, skiing, skills * fish, fished, fishing, fisher, fishxsdfe Each time, with QueryStemming on, skills and fishxsdfe come out in the search results. Is it what I should expect? "Skills", maybe but "fishxsdfe"? Do you know how it works? or have a good example that would help me to understand? Thanks! -- François Charbonnier, Bibl. prof. / Chef de produits Tél. : (888) 604-2627 <mailto:francois.charbonnier@inLibro.com> francois.charbonnier@inLibro.com inLibro | pour esprit libre | <http://www.inLibro.com> www.inLibro.com
Hi I had always thought stemming was made by Zebra, and only in english! In fact the algorithm for french language is here: http://snowball.tartarus.org/algorithms/french/stemmer.html (Lingua::Stem::Snowball is a Perl interface to the C version of the Snowball stemmers) Mathieu Saby Le 27/08/2014 10:22, David Cook a écrit :
Hi Francois:
I wrote an email earlier on my tablet, but not 100% sure if it got sent. In any case, I'm writing again now!
You'll want to look at C4::Search::_build_stemmed_operand().
Zebra doesn't actually do any stemming itself. If you read through the Zebra docs (if you're masochistic), you'll notice that they say explicitly that Zebra doesn't do any stemming, but that you can do stemming (using a stemmer like Snowball) while building a query. That's exactly what we do in Koha.
The Perl module that does the stemming is Lingua::Stem::Snowball.
However, you might notice that your query's operands aren't always stemmed properly. I haven't looked in a while, but I think it's because we don't build our queries very well at all (when not using QueryParser).
If you want to understand why you're getting "skills" and "fishxsdfe" in your results, I would suggest running some tests ( using "Data::Dumper" and "warn" ) so that you can see your query as it is built.
I have a lot of work I want to do on C4::Search::buildQuery, but just don't have the time :/.
Unfortunately, at the moment, there is no stemming when using the QueryParser. However, fortunately, using Lingua::Stem::Snowball with QueryParser would be really really easy. I think that I've written a note on how to do that somewhere on Bugzilla or maybe on Trello...
I hope that helps! Feel free to send me an email or shout at me on IRC if you want any clarification. I know I probably didn't make it any clearer but hopefully this might help you on your path to understanding.
David Cook
Systems Librarian
Prosentient Systems
72/330 Wattle St, Ultimo, NSW 2007
*From:*koha-devel-bounces@lists.koha-community.org [mailto:koha-devel-bounces@lists.koha-community.org] *On Behalf Of *Francois Charbonnier *Sent:* Wednesday, 27 August 2014 2:09 AM *To:* koha-devel@lists.koha-community.org *Subject:* [Koha-devel] Stemming and zebra
Hello,
I have tested the QueryStemming system preference on Koha 3.14 (my local installation) and I'm wondering, does zebra just right truncate the words or is there an algorithm to find the stems?
I use ICU and I have enabled "QueryWeightFields". I don't have automatic truncation or fuzzy search on. I use these words for my tests:
* ski, skiing, skills * fish, fished, fishing, fisher, fishxsdfe
Each time, with QueryStemming on, skills and fishxsdfe come out in the search results. Is it what I should expect? "Skills", maybe but "fishxsdfe"?
Do you know how it works? or have a good example that would help me to understand?
Thanks!
--
François Charbonnier, Bibl. prof. / Chef de produits
Tél. : (888) 604-2627 francois.charbonnier@inLibro.com <mailto:francois.charbonnier@inLibro.com>
inLibro| pour esprit libre |www.inLibro.com <http://www.inLibro.com>
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
Hi Mathieu: I think many of us think certain things happen in Zebra when they actually happen in Koha before the query ever reaches Zebra ;). As for stemming, theoretically the language obtained via C4::Templates::getlanguage($cgi, 'intranet'); should filter down into the Snowball stemming. If it isnt working in French, it might be because the right locale isnt being passed to Snowball correctly. Thats very possible as I think were using arbitrary language codes rather than standard locales in some cases. It looks like there is a fallback to English in C4::Templates::getlanguage() as well. If its not working for French, it probably just needs a tweak! Yeah, I first heard about Snowball when reading through Zebra docs, and I was pleasantly surprised when I saw that Lingua::Stem::Snowball existed as a Perl interface for the C program. David Cook Systems Librarian Prosentient Systems 72/330 Wattle St, Ultimo, NSW 2007 From: koha-devel-bounces@lists.koha-community.org [mailto:koha-devel-bounces@lists.koha-community.org] On Behalf Of Mathieu Saby Sent: Wednesday, 27 August 2014 7:30 PM To: koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] Stemming and zebra Hi I had always thought stemming was made by Zebra, and only in english! In fact the algorithm for french language is here: http://snowball.tartarus.org/algorithms/french/stemmer.html (Lingua::Stem::Snowball is a Perl interface to the C version of the Snowball stemmers) Mathieu Saby Le 27/08/2014 10:22, David Cook a écrit : Hi Francois: I wrote an email earlier on my tablet, but not 100% sure if it got sent. In any case, Im writing again now! Youll want to look at C4::Search::_build_stemmed_operand(). Zebra doesnt actually do any stemming itself. If you read through the Zebra docs (if youre masochistic), youll notice that they say explicitly that Zebra doesnt do any stemming, but that you can do stemming (using a stemmer like Snowball) while building a query. Thats exactly what we do in Koha. The Perl module that does the stemming is Lingua::Stem::Snowball. However, you might notice that your querys operands arent always stemmed properly. I havent looked in a while, but I think its because we dont build our queries very well at all (when not using QueryParser). If you want to understand why youre getting skills and fishxsdfe in your results, I would suggest running some tests ( using Data::Dumper and warn ) so that you can see your query as it is built. I have a lot of work I want to do on C4::Search::buildQuery, but just dont have the time :/. Unfortunately, at the moment, there is no stemming when using the QueryParser. However, fortunately, using Lingua::Stem::Snowball with QueryParser would be really really easy. I think that Ive written a note on how to do that somewhere on Bugzilla or maybe on Trello I hope that helps! Feel free to send me an email or shout at me on IRC if you want any clarification. I know I probably didnt make it any clearer but hopefully this might help you on your path to understanding. David Cook Systems Librarian Prosentient Systems 72/330 Wattle St, Ultimo, NSW 2007 From: koha-devel-bounces@lists.koha-community.org <mailto:koha-devel-bounces@lists.koha-community.org> [mailto:koha-devel-bounces@lists.koha-community.org] On Behalf Of Francois Charbonnier Sent: Wednesday, 27 August 2014 2:09 AM To: koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org> Subject: [Koha-devel] Stemming and zebra Hello, I have tested the QueryStemming system preference on Koha 3.14 (my local installation) and I'm wondering, does zebra just right truncate the words or is there an algorithm to find the stems? I use ICU and I have enabled "QueryWeightFields". I don't have automatic truncation or fuzzy search on. I use these words for my tests: ski, skiing, skills fish, fished, fishing, fisher, fishxsdfe Each time, with QueryStemming on, skills and fishxsdfe come out in the search results. Is it what I should expect? "Skills", maybe but "fishxsdfe"? Do you know how it works? or have a good example that would help me to understand? Thanks! -- François Charbonnier, Bibl. prof. / Chef de produits Tél. : (888) 604-2627 <mailto:francois.charbonnier@inLibro.com> francois.charbonnier@inLibro.com inLibro | pour esprit libre | <http://www.inLibro.com> www.inLibro.com _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
Hi, Thanks Brooke for the support. I did read the rfc by the way but it's for queryparser which I'm not using now. I wish I'll find time to test it! Thanks David, I haven't read the whole zebra documention (ahah, sounds crazy) but did some researches. I had the impression zebra didn't manage stemming but it wasn't clear. So thank you for your answer. It will definitely help me to understand! Thanks Mathieu for the link. I'll have a look. For sure! Have a good day ! :^) François François Charbonnier, Bibl. prof. / Chef de produits Tél. : (888) 604-2627 francois.charbonnier@inLibro.com <mailto:francois.charbonnier@inLibro.com> inLibro | pour esprit libre | www.inLibro.com <http://www.inLibro.com> Le 2014-08-27 05:48, David Cook a écrit :
Hi Mathieu:
I think many of us think certain things happen in Zebra when they actually happen in Koha before the query ever reaches Zebra ;).
As for stemming, theoretically the language obtained via “C4::Templates::getlanguage($cgi, 'intranet');” should filter down into the Snowball stemming. If it isn’t working in French, it might be because the right locale isn’t being passed to Snowball correctly. That’s very possible as I think we’re using arbitrary language codes rather than standard locales in some cases. It looks like there is a fallback to English in C4::Templates::getlanguage() as well. If it’s not working for French, it probably just needs a tweak!
Yeah, I first heard about Snowball when reading through Zebra docs, and I was pleasantly surprised when I saw that Lingua::Stem::Snowball existed as a Perl interface for the C program.
David Cook
Systems Librarian
Prosentient Systems
72/330 Wattle St, Ultimo, NSW 2007
*From:*koha-devel-bounces@lists.koha-community.org [mailto:koha-devel-bounces@lists.koha-community.org] *On Behalf Of *Mathieu Saby *Sent:* Wednesday, 27 August 2014 7:30 PM *To:* koha-devel@lists.koha-community.org *Subject:* Re: [Koha-devel] Stemming and zebra
Hi
I had always thought stemming was made by Zebra, and only in english!
In fact the algorithm for french language is here: http://snowball.tartarus.org/algorithms/french/stemmer.html
(Lingua::Stem::Snowball is a Perl interface to the C version of the Snowball stemmers)
Mathieu Saby
Le 27/08/2014 10:22, David Cook a écrit :
Hi Francois:
I wrote an email earlier on my tablet, but not 100% sure if it got sent. In any case, I’m writing again now!
You’ll want to look at C4::Search::_build_stemmed_operand().
Zebra doesn’t actually do any stemming itself. If you read through the Zebra docs (if you’re masochistic), you’ll notice that they say explicitly that Zebra doesn’t do any stemming, but that you can do stemming (using a stemmer like Snowball) while building a query. That’s exactly what we do in Koha.
The Perl module that does the stemming is Lingua::Stem::Snowball.
However, you might notice that your query’s operands aren’t always stemmed properly. I haven’t looked in a while, but I think it’s because we don’t build our queries very well at all (when not using QueryParser).
If you want to understand why you’re getting “skills” and “fishxsdfe” in your results, I would suggest running some tests ( using “Data::Dumper” and “warn” ) so that you can see your query as it is built.
I have a lot of work I want to do on C4::Search::buildQuery, but just don’t have the time :/.
Unfortunately, at the moment, there is no stemming when using the QueryParser. However, fortunately, using Lingua::Stem::Snowball with QueryParser would be really really easy. I think that I’ve written a note on how to do that somewhere on Bugzilla or maybe on Trello…
I hope that helps! Feel free to send me an email or shout at me on IRC if you want any clarification. I know I probably didn’t make it any clearer but hopefully this might help you on your path to understanding.
David Cook
Systems Librarian
Prosentient Systems
72/330 Wattle St, Ultimo, NSW 2007
*From:*koha-devel-bounces@lists.koha-community.org <mailto:koha-devel-bounces@lists.koha-community.org> [mailto:koha-devel-bounces@lists.koha-community.org] *On Behalf Of *Francois Charbonnier *Sent:* Wednesday, 27 August 2014 2:09 AM *To:* koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org> *Subject:* [Koha-devel] Stemming and zebra
Hello,
I have tested the QueryStemming system preference on Koha 3.14 (my local installation) and I'm wondering, does zebra just right truncate the words or is there an algorithm to find the stems?
I use ICU and I have enabled "QueryWeightFields". I don't have automatic truncation or fuzzy search on. I use these words for my tests:
ski, skiing, skills
fish, fished, fishing, fisher, fishxsdfe
Each time, with QueryStemming on, skills and fishxsdfe come out in the search results. Is it what I should expect? "Skills", maybe but "fishxsdfe"?
Do you know how it works? or have a good example that would help me to understand?
Thanks!
--
François Charbonnier, Bibl. prof. / Chef de produits
Tél. : (888) 604-2627 francois.charbonnier@inLibro.com <mailto:francois.charbonnier@inLibro.com>
inLibro| pour esprit libre |www.inLibro.com <http://www.inLibro.com>
_______________________________________________
Koha-devel mailing list
Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org>
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website :http://www.koha-community.org/
git :http://git.koha-community.org/
bugs :http://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
participants (3)
-
David Cook -
Francois Charbonnier -
Mathieu Saby