[Bug 20151] New: Search is broken when stemming has no language
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20151 Bug ID: 20151 Summary: Search is broken when stemming has no language Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: minor Priority: P5 - low Component: Searching Assignee: koha-bugs@lists.koha-community.org Reporter: fridolin.somers@biblibre.com QA Contact: testopia@bugs.koha-community.org When stemming is enable in catalog searching, C4::Search::_build_stemmed_operand will transform operand into stemmed operand using stemmer Lingua::Stem::Snowball with a specified language. This stemmer returns undef is no language is defined. In classic catalog search, current language is used. But in other pages acqui/neworderbiblio.pl and cataloguing/addbooks.pl no language is defined so operand is empty. Any search returns entire catalog. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20151 Fridolin SOMERS <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |fridolin.somers@biblibre.co |ity.org |m -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20151 Fridolin SOMERS <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|--- |Trivial patch Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20151 --- Comment #1 from Fridolin SOMERS <fridolin.somers@biblibre.com> --- Created attachment 71285 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=71285&action=edit Bug 20151: always use current language for stemming When stemming is enable in catalog searching, C4::Search::_build_stemmed_operand will transform operand into stemmed operand using stemmer Lingua::Stem::Snowball with a specified language. This stemmer returns undef is no language is defined. In classic catalog search, current language is used. But in other pages acqui/neworderbiblio.pl and cataloguing/addbooks.pl no language is defined so operand is empty. Any search returns entire catalog. This patch corrects by returning operand without change if no langage is defined in C4::Search::_build_stemmed_operand. And uses current langage in pages acqui/neworderbiblio.pl and cataloguing/addbooks.pl. Test plan : 1) Enable system preferences QueryStemming and QueryWeightFields 2) Disable system preferences QueryAutoTruncate, QueryFuzzy and UseQueryParser 3) Perform an search in catalog (without index) that uses the stemming, for example "historical" will search "histor*" 4) Go to "Cataloging" 5) Perform a search on same word in "Cataloging search" 6) Without patch you have entire catalog, with patch you have correct number of results 7) Go to aquisition on a basket and click "Add to basket" 8) Perform search in "From an existing record" 9) Without patch you have entire catalog, with patch you have correct number of results -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20151 jmbroust <jean-manuel.broust@univ-lyon2.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA CC| |jean-manuel.broust@univ-lyo | |n2.fr --- Comment #2 from jmbroust <jean-manuel.broust@univ-lyon2.fr> --- Sorry, I can't reproduce the problem on sandbox 8. One point on test plan needs an explanation : QueryAutoTruncate must be turned to 'automaticaly' or to "only if * is added' ? Whatever I do, I never have the whole catalog as number of results. I searched with the word 'france' and I had the same number of results or less results in cataloging search vs catalog search. Am I missing something ? or problem with sandbox ? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20151 Fridolin SOMERS <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff --- Comment #3 from Fridolin SOMERS <fridolin.somers@biblibre.com> --- (In reply to jmbroust from comment #2)
Sorry, I can't reproduce the problem on sandbox 8. One point on test plan needs an explanation : QueryAutoTruncate must be turned to 'automaticaly' or to "only if * is added' ?
2) Disable system preference QueryAutoTruncate I mean "only if * is added" -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20151 Fridolin SOMERS <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #71285|0 |1 is obsolete| | --- Comment #4 from Fridolin SOMERS <fridolin.somers@biblibre.com> --- Created attachment 78994 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=78994&action=edit Bug 20151: always use current language for stemming When stemming is enabled, in catalog searching "C4::Search::_build_stemmed_operand" will transform query operand into stemmed operand using stemmer Lingua::Stem::Snowball with a specified language. This stemmer returns undef stemmed operand if no language is defined. In main catalog search (catalogue/search.pl) current language is used. But in other pages "acqui/neworderbiblio.pl" and "cataloguing/addbooks.pl" no language is defined so stemmed operand is empty and so stemming is not applied. This patch corrects by returning in "C4::Search::_build_stemmed_operand" operand without change if no langage is defined. And uses current langage in pages "acqui/neworderbiblio.pl" and "cataloguing/addbooks.pl" so all catalog search uses stemming. Test plan : 1) Enable system preferences QueryStemming and QueryWeightFields 2) Disable system preferences QueryAutoTruncate, QueryFuzzy and UseQueryParser 3) Go to intranet main page and click on "Search the catalog" tab 4) Perform a search (without index) that uses the stemming, for example searching for "years" will also match "year" 5) Note how many results you get, for example "year" gets 24 results and "years" gets 24 results 6) Go to "Cataloging" module 7) Perform a search on same word in "Cataloging search" tab 8) Note how many results you get 9) Without patch you get fewer results than first search (step 5) because stemming is not applied, for example "year" gets 11 results and "years" gets 15 results 10) With patch you get the same results as first search (step 5) because stemming is applied, for example "year" and "years" gets 24 results 11) Same tests in aquisition module 12) On a basket, click "Add to basket" and perform a search in "From an existing record" -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20151 --- Comment #5 from Fridolin SOMERS <fridolin.somers@biblibre.com> --- I changed test plan after tests with David Nind on new koha-testing-docker. Example with year/years works in this environment. It may be easier for QA. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20151 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20151 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #78994|0 |1 is obsolete| | --- Comment #6 from David Nind <david@davidnind.com> --- Created attachment 78997 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=78997&action=edit Bug 20151: always use current language for stemming When stemming is enabled, in catalog searching "C4::Search::_build_stemmed_operand" will transform query operand into stemmed operand using stemmer Lingua::Stem::Snowball with a specified language. This stemmer returns undef stemmed operand if no language is defined. In main catalog search (catalogue/search.pl) current language is used. But in other pages "acqui/neworderbiblio.pl" and "cataloguing/addbooks.pl" no language is defined so stemmed operand is empty and so stemming is not applied. This patch corrects by returning in "C4::Search::_build_stemmed_operand" operand without change if no langage is defined. And uses current langage in pages "acqui/neworderbiblio.pl" and "cataloguing/addbooks.pl" so all catalog search uses stemming. Test plan : 1) Enable system preferences QueryStemming and QueryWeightFields 2) Disable system preferences QueryAutoTruncate, QueryFuzzy and UseQueryParser 3) Go to intranet main page and click on "Search the catalog" tab 4) Perform a search (without index) that uses the stemming, for example searching for "years" will also match "year" 5) Note how many results you get, for example "year" gets 24 results and "years" gets 24 results 6) Go to "Cataloging" module 7) Perform a search on same word in "Cataloging search" tab 8) Note how many results you get 9) Without patch you get fewer results than first search (step 5) because stemming is not applied, for example "year" gets 11 results and "years" gets 15 results 10) With patch you get the same results as first search (step 5) because stemming is applied, for example "year" and "years" gets 24 results 11) Same tests in aquisition module 12) On a basket, click "Add to basket" and perform a search in "From an existing record" Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20151 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA CC| |martin.renvoize@ptfs-europe | |.com --- Comment #7 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- I'd really love to see some tests added for this.. but as it's inside a C4 area I can't justifiably fail qa for that. Generally, the code makes sense and doesn't appear to cause any regressions on testing and fixes a valid issue. Passing QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20151 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #78997|0 |1 is obsolete| | --- Comment #8 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 80253 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=80253&action=edit Bug 20151: always use current language for stemming When stemming is enabled, in catalog searching "C4::Search::_build_stemmed_operand" will transform query operand into stemmed operand using stemmer Lingua::Stem::Snowball with a specified language. This stemmer returns undef stemmed operand if no language is defined. In main catalog search (catalogue/search.pl) current language is used. But in other pages "acqui/neworderbiblio.pl" and "cataloguing/addbooks.pl" no language is defined so stemmed operand is empty and so stemming is not applied. This patch corrects by returning in "C4::Search::_build_stemmed_operand" operand without change if no langage is defined. And uses current langage in pages "acqui/neworderbiblio.pl" and "cataloguing/addbooks.pl" so all catalog search uses stemming. Test plan : 1) Enable system preferences QueryStemming and QueryWeightFields 2) Disable system preferences QueryAutoTruncate, QueryFuzzy and UseQueryParser 3) Go to intranet main page and click on "Search the catalog" tab 4) Perform a search (without index) that uses the stemming, for example searching for "years" will also match "year" 5) Note how many results you get, for example "year" gets 24 results and "years" gets 24 results 6) Go to "Cataloging" module 7) Perform a search on same word in "Cataloging search" tab 8) Note how many results you get 9) Without patch you get fewer results than first search (step 5) because stemming is not applied, for example "year" gets 11 results and "years" gets 15 results 10) With patch you get the same results as first search (step 5) because stemming is applied, for example "year" and "years" gets 24 results 11) Same tests in aquisition module 12) On a basket, click "Add to basket" and perform a search in "From an existing record" Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20151 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nick@bywatersolutions.com Status|Passed QA |Pushed to Master --- Comment #9 from Nick Clemens <nick@bywatersolutions.com> --- Awesome work all! Pushed to master for 18.11 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20151 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to Master |Pushed to Stable --- Comment #10 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Pushed to 18.05.x for 18.05.05 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20151 --- Comment #11 from Fridolin SOMERS <fridolin.somers@biblibre.com> --- Pushed to 17.11.x for 17.11.11 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20151 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to Stable |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org