[Bug 13414] New: Enable right truncation with search by relevance
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13414 Bug ID: 13414 Summary: Enable right truncation with search by relevance Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Searching Assignee: gmcharlt@gmail.com Reporter: koha@univ-lyon3.fr QA Contact: testopia@bugs.koha-community.org Created attachment 34199 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=34199&action=edit Enable right truncation with search by relevance This patch slightly modifies build_weighted_query function (Search.pm) that defines the ranking levels for a search by relevance in order to make right truncation possible when QueryWeightFields syspref is activated (even ranked with little weight) it modifies the type of this syspref from boolean to choice in order to add this possiblity as a third option : Search by relevance with right truncation. Default remains Search by relevance (without truncation). -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13414 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #34199|0 |1 is patch| | Attachment #34199|application/mbox |text/plain mime type| | CC| |katrin.fischer@bsz-bw.de -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13414 --- Comment #1 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I am a bit worried about the database update - wouldn't that set your syspref to 1 even if the it was 0 before? Maybe just changing the values that need to be changed would work? -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13414 Koha Team Lyon 3 <koha@univ-lyon3.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #34199|0 |1 is obsolete| | CC| |koha@univ-lyon3.fr --- Comment #2 from Koha Team Lyon 3 <koha@univ-lyon3.fr> --- Created attachment 34285 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=34285&action=edit Bug 13414 - Enable automatic right truncation while searching by relevance Oups ! I fear you are absolutely right. patch corrected. Moreover some more precisions were missing : By now, relevance don't work when Automatic truncation is activated : the results are sorted by biblionumber ascending. However truncation (left and right) is possible with relevance activated when QueryAutoTruncate is set on manual option ( this means you have to add * character to your keyword). This is the automatic right truncation that this patch brings combined with relevance. These two options cannot get along : the choice is Every where manual truncation versus Automatic right truncation. I just add right truncation as a trial because it seems to match the essential and common needs : getting the plural and very closed terms. I fear going further would be a bit of tuning. TEST PLAN : 1) Apply patch. Set QueryWeightFields syspref on default option (relevance without truncation) and QueryAutoTruncate on automatic position. 2) choose a keysearch not too much common so as to get only a small batch of results : note the number of results. Note that results are sorted by biblionumber ascending and not by relevance. To check, pass the cursor over the titles.That displays the link address in left down corner of active window : biblionumber is visible at the end of the address. 3) redo the search adding an s plural to your keysearch (or anything so as keeping keysearch as radical) : note the number of results. this can be higher than previous ones. 4) Among these last results, try to find a record where only the plural form is present (checking in Marc tab). Call it recordTest. 5) Turn QueryWeightFields syspref on right truncation option, redo 1 and note the results number. It's not any more sorted by biblionumber. Little more by relevance( little more because relevance in koha may be surprising !). There could be results where the plural form of keyword had been found (and the singular one is not present), more possibly at the end of the list. To be sure, look for recordTest record. 6) redo 3 and note the number of results. this one should be smaller (these results must be included in previous one). Of course, you can redo the tests with QueryAutoTruncate in "manual" position adding a right * and compare the results. Olivier Crouzet -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13414 Koha Team Lyon 3 <koha@univ-lyon3.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff --- Comment #3 from Koha Team Lyon 3 <koha@univ-lyon3.fr> --- Well, painfully start for this patch, my assertion about manual truncation was wrong. So consider this comment cancel the previous one (comment 2) : Moreover some more precisions were missing : By now, relevance seems to not work at all when Automatic truncation is activated or manual Truncation is used (* wilcard ) : the results are sorted by biblionumber ascending. It's automatic right truncation combined with relevance that this patch brings. I just add right truncation as a trial because it seems to match the essential and common needs : getting the plural and very closed terms. I fear going further would be a bit of tuning. By caution, I add it as a third option but maybe, it could replace the existing enable one. TEST PLAN : 1) Apply patch. Set QueryWeightFields syspref on default option (relevance without truncation). 2) choose a keysearch not too much common so as to get only a small batch of results : note the number of results. Note that results are sorted by biblionumber ascending and not by relevance. To check, pass the cursor over the titles.That displays the link address in left down corner of active window : biblionumber is visible at the end of the address. 3) redo the search adding an s plural to your keysearch (or anything so as keeping keysearch as radical) : note the number of results. this can be higher than previous ones. 4) Among these last results, try to find a record where only the plural form is present (checking in Marc tab). Call it recordTest. 5) Turn QueryWeightFields syspref on right truncation option, redo 1 and note the results number. It's not any more sorted by biblionumber. Little more by relevance( little more because relevance in koha is far from beeing perfect : for example high ranking on exact title doesn't make effects). There could be results where the plural form of keyword had been found (and the singular one is not present), more possibly at the end of the list. To be sure, look for recordTest record. 6) redo 3 and note the number of results. this one should be smaller (these results must be included in previous one). -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13414 sandboxes@biblibre.com <sandboxes@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off CC| |sandboxes@biblibre.com --- Comment #4 from sandboxes@biblibre.com <sandboxes@biblibre.com> --- Patch tested with a sandbox, by kohateam rennes 2 <koha@listes.univ-rennes2.fr> -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13414 sandboxes@biblibre.com <sandboxes@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #34285|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13414 --- Comment #5 from sandboxes@biblibre.com <sandboxes@biblibre.com> --- Created attachment 34316 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=34316&action=edit Bug 13414 - Enable automatic right truncation while searching by relevance This makes slight modification in build_weighted_query function in order to make automatic right truncation possible when QueryWeightFields syspref activated (even little ranked) Modify this syspref type from boolean to choice in order to add this possiblity as a third option : Search by relevance with automatic right truncation. Signed-off-by: kohateam rennes 2 <koha@listes.univ-rennes2.fr> -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13414 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au, | |martin.renvoize@ptfs-europe | |.com -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13414 --- Comment #6 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- David, as the current zebra guru, any chance you could give these patches a glance over? -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13414 --- Comment #7 from David Cook <dcook@prosentient.com.au> --- Comment on attachment 34316 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=34316 Bug 13414 - Enable automatic right truncation while searching by relevance Review of attachment 34316: --> (http://bugs.koha-community.org/bugzilla3/page.cgi?id=splinter.html&bug=13414&attachment=34316) ----------------------------------------------------------------- ::: C4/Search.pm @@ +1030,5 @@
#$weighted_query .=" or kw,wrdl,r5=\"$operand\""; # word list any $weighted_query .= " or wrdl,fuzzy,r8=\"$operand\"" if $fuzzy_enabled; # add fuzzy, word list + if ( $stemming and $stemmed_operand ) { + $weighted_query .= " or wrdl,right-Truncation,r9=\"$stemmed_operand\""
No functional difference... would be better to have this in a separate patch. @@ +1620,4 @@
# Apply Truncation if ( scalar(@$righttruncated) + scalar(@$lefttruncated) + + scalar(@$rightlefttruncated) > 0 and $weight_fields < 2 )
This will disable all truncation except for right truncation (which will be forced), if "QueryWeightFields" is set to 2. I don't love this, but seeing as truncation currently disables "QueryWeightFields", it might be fine. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13414 --- Comment #8 from David Cook <dcook@prosentient.com.au> --- (In reply to Martin Renvoize from comment #6)
David, as the current zebra guru, any chance you could give these patches a glance over?
Honestly, "buildQuery" and "_build_weighted_query" are so badly written that any change is probably an improvement :p. While I think this patch is a workaround, I think it's probably a reasonable enough workaround. -- What we really need to do is have functions that test for "QueryStemming", "QueryFuzzy", "QueryWeightFields", "QueryAutoTruncate", manual truncation, and index type THEN add the appropriate qualifiers. Rather than the bizarre mishmash that we have currently. Ideally, QueryParser should do this, but it hasn't been finished yet. Personally, I want to re-write buildQuery, but haven't had the time. I'd like to use an object or a hash for each atomic query, process them (according to our sysprefs and to make them valid CCL), and then build those atomic queries into a the complete query. It's basically what we do now, except you can't do everything to a query when realistically you should be able to do so. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13414 --- Comment #9 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- David, thx for your comments. Could you give this a second sign-off? -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13414 --- Comment #10 from David Cook <dcook@prosentient.com.au> --- (In reply to Katrin Fischer from comment #9)
David, thx for your comments. Could you give this a second sign-off?
Yep, but not today. I'll put it on my to do list for later this week! -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13414 --- Comment #11 from David Cook <dcook@prosentient.com.au> --- Comment on attachment 34316 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=34316 Bug 13414 - Enable automatic right truncation while searching by relevance Review of attachment 34316: --> (http://bugs.koha-community.org/bugzilla3/page.cgi?id=splinter.html&bug=13414&attachment=34316) ----------------------------------------------------------------- ::: installer/data/mysql/sysprefs.sql @@ +343,4 @@
('QueryAutoTruncate','1',NULL,'If ON, query truncation is enabled by default','YesNo'), ('QueryFuzzy','1',NULL,'If ON, enables fuzzy option for searches','YesNo'), ('QueryStemming','1',NULL,'If ON, enables query stemming','YesNo'), +('QueryWeightFields','1','0|1|2','Enable field weighting research','Choice'),
"Enable field weighting research"? What does research mean in this context? -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13414 --- Comment #12 from Koha Team Lyon 3 <koha@univ-lyon3.fr> --- Yes, it would be more exact to say "Enable field weighting sorting" or to keep the initial label "enables field weighting" -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13414 --- Comment #13 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- David, Team Lyon3, how do we proceed here? Would you be ok with me changing the sysprefs.sql file back to the old label? -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13414 --- Comment #14 from Koha Team Lyon 3 <koha@univ-lyon3.fr> --- (In reply to Katrin Fischer from comment #13)
David, Team Lyon3, how do we proceed here? Would you be ok with me changing the sysprefs.sql file back to the old label?
it's ok for me Olivier Crouzet -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13414 --- Comment #15 from David Cook <dcook@prosentient.com.au> --- (In reply to Katrin Fischer from comment #13)
David, Team Lyon3, how do we proceed here? Would you be ok with me changing the sysprefs.sql file back to the old label?
It doesn't really matter too much what's in sysprefs.sql, since it won't show up in the web UI, but I figure keeping the original label is probably the best idea. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13414 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|gmcharlt@gmail.com |koha@univ-lyon3.fr QA Contact|testopia@bugs.koha-communit |m.de.rooy@rijksmuseum.nl |y.org | CC| |m.de.rooy@rijksmuseum.nl -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13414 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #34316|0 |1 is obsolete| | --- Comment #16 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 38024 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=38024&action=edit Bug 13414 - Enable automatic right truncation while searching by relevance This makes slight modification in build_weighted_query function in order to make automatic right truncation possible when QueryWeightFields syspref activated (even little ranked) Modify this syspref type from boolean to choice in order to add this possiblity as a third option : Search by relevance with automatic right truncation. Signed-off-by: kohateam rennes 2 <koha@listes.univ-rennes2.fr> -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13414 --- Comment #17 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Just a rebase yet.. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13414 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Failed QA --- Comment #18 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- QA Comment: Thanks for sending your patch. I have been thinking about your patch a while. But in the end it seems to me that we should not be mixing the features ranking and truncation in one syspref. Search.pm is already hard to maintain. A crucial piece of your code: - $weighted_query .= " or wrdl,r9=\"$operand\""; - + } elsif ( $weight_fields == 2 ) { + $weighted_query .= " or wrdl,right-Truncation,r9=\"$operand\""; + } else { + $weighted_query .= " or wrdl,r9=\"$operand\""; Question: Could you think of another way to determine when we want right truncation in the deleted first line? Do we really need an adjusted pref? Since this is a complicated patch (changing Search.pm), I would also like a thorough test plan. There is no test plan now. And I would appreciate to hear from the one who signed off: What did you exactly test? Small stuff: [1] Add type choice in the pref file. [2] There is no print statement in the db rev. Moving this to Failed QA for now. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13414 Fridolin SOMERS <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13414 --- Comment #19 from Fridolin SOMERS <fridolin.somers@biblibre.com> --- Syspref could be more friendly with strings instead of numbers. My 2c -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org