[Bug 25900] New: Double quotes breaks search with Zebra and QueryWeightFields
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25900 Bug ID: 25900 Summary: Double quotes breaks search with Zebra and QueryWeightFields Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Searching - Zebra Assignee: koha-bugs@lists.koha-community.org Reporter: fridolin.somers@biblibre.com Search with Zebra and system preference QueryWeightFields on builds a query with surrounding double quotes : q=tree => q="tree". Therefore existing double quotes are replaced by a space : q="tree" => q=" tree ". Looks like the leading space breaks the search, 0 results. Experimented on Ubuntu Xenial and Bionic with Indexdata repository so Zebra version is 2.1.4. It might be specific to this version. You can reproduce with direct CCL query (regardless QueryWeightFields) : https://demo.biblibre.com/cgi-bin/koha/opac-search.pl?idx=&q=ccl%3Dti%2Cphr%3D%22centre+arch%C3%A9ologique%22 https://demo.biblibre.com/cgi-bin/koha/opac-search.pl?idx=&q=ccl%3Dti%2Cphr%3D%22+centre+arch%C3%A9ologique%22 -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25900 Fridolin SOMERS <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |fridolin.somers@biblibre.co |ity.org |m Status|NEW |ASSIGNED -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25900 Fridolin SOMERS <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Needs Signoff Patch complexity|--- |Trivial patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25900 --- Comment #1 from Fridolin SOMERS <fridolin.somers@biblibre.com> --- Created attachment 106432 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=106432&action=edit Bug 25900: trim spaces in search with Zebra and QueryWeightFields Search with Zebra and system preference QueryWeightFields on builds a query with surrounding double quotes : q=tree => q="tree". Therefore existing double quotes are replaced by a space : q="tree" => q=" tree ". Looks like the leading space breaks the search, 0 results. Experimented on Ubuntu Xenial and Bionic with Indexdata repository so Zebra version is 2.1.4. It might be specific to this version. Patch corrects by adding trim on operand after transforming quotes into spaces. Test plan : 1) Use Zebra searchengine and enable system preference QueryWeightFields 2) Go to advanced search 3) Search on title two words you know exists in title surrounded by double quotes, like "Order of the Phoenix" 4) With patch you get results, no results without patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25900 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25900 Theodoros Theodoropoulos <theod@lib.auth.gr> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |theod@lib.auth.gr -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25900 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA --- Comment #2 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Hi Fridolin, I just tested this on master without your patch and it search works fine for me - could another pref be at play here? I tried simple searches on staff interface like: "simon of" Using the standard MARC21 sample data. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25900 --- Comment #3 from Fridolin SOMERS <fridolin.somers@biblibre.com> --- Which Zebra version please ? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25900 --- Comment #4 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- kohadevbox: Zebra version: Zebra 2.0.59 (C) 1 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25900 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25900 --- Comment #5 from Fridolin SOMERS <fridolin.somers@biblibre.com> --- Oh indeed, even Debian Buster has Zebra 2.0.59. You can use indexdata repo to upgrade the version : http://ftp.indexdata.dk/debian/ -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25900 --- Comment #6 from David Cook <dcook@prosentient.com.au> --- Frido: Are you using CHR or ICU with Zebra? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25900 --- Comment #7 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I've reset the status, so maybe someone with a newer version can give it a go. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25900 --- Comment #8 from Fridolin SOMERS <fridolin.somers@biblibre.com> --- (In reply to David Cook from comment #6)
Frido: Are you using CHR or ICU with Zebra?
Always ICU -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25900 --- Comment #9 from David Cook <dcook@prosentient.com.au> --- (In reply to Fridolin SOMERS from comment #8)
(In reply to David Cook from comment #6)
Frido: Are you using CHR or ICU with Zebra?
Always ICU
That's what I thought. I suspect that Katrin and others are probably using CHR, so that might also explain the discrepancy in results. I haven't looked into the patches though. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25900 --- Comment #10 from David Cook <dcook@prosentient.com.au> --- Using koha-testing-docker w/o patch using CHR in /etc/koha/zebradb/etc/default.idx: QueryAutoTruncate: Query: test Request: @attrset Bib-1 @attr 1=1016 @attr 4=6 @attr 5=1 test Query: "test" Request: @attrset Bib-1 @attr 1=1016 @attr 4=6 @attr 5=1 @attr 2=102 test QueryWeightFields: Query: test Request: @attrset Bib-1 @or @or @or @or @or @or @attr 1=36 @attr 4=1 @attr 6=3 @attr 9=32 @attr 2=102 test @attr 1=4 @attr 4=1 @attr 6=3 @attr 9=28 @attr 2=102 test @attr 1= 36 @attr 4=1 @attr 9=26 @attr 2=102 test @attr 1=4 @attr 4=6 @attr 9=24 @attr 2=102 test @attr 4=6 @attr 5=103 @attr 9=16 @attr 2=102 test @attr 4=6 @attr 5=1 @attr 9=14 @attr 2=102 "test? " @attr 4=6 @attr 9=14 @attr 2=102 test Query: "test" Request: @attrset Bib-1 @or @or @or @or @or @or @attr 1=36 @attr 4=1 @attr 6=3 @attr 9=32 @attr 2=102 " test " @attr 1=4 @attr 4=1 @attr 6=3 @attr 9=28 @attr 2=102 " test " @att r 1=36 @attr 4=1 @attr 9=26 @attr 2=102 " test " @attr 1=4 @attr 4=6 @attr 9=24 @attr 2=102 " test " @attr 4=6 @attr 5=103 @attr 9=16 @attr 2=102 " test " @attr 4=6 @attr 5=1 @attr 9=14 @attr 2=102 "test? " @attr 4=6 @attr 9=14 @attr 2=1 02 " test " -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25900 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA --- Comment #11 from David Cook <dcook@prosentient.com.au> --- Using koha-testing-docker w/o patch using ICU in /etc/koha/zebradb/etc/default.idx: For "index w", replace "charmap word-phrase-utf.chr" with "icuchain words-icu.xml" For "index p", replace "charmap word-phrase-utf.chr" with "icuchain phrases-icu.xml" koha-rebuild-zebra -f -b -v kohadev QueryAutoTruncate: Query: test Request: Search biblios OK 3 1 1+0 RPN @attrset Bib-1 @attr 1=1016 @attr 4=6 @attr 5=1 test Query: "test" Request: Search biblios OK 3 1 1+0 RPN @attrset Bib-1 @attr 1=1016 @attr 4=6 @attr 5=1 test QueryWeightFields: Query: test Request: Search biblios ERROR 120 1 1+0 RPN @attrset Bib-1 @or @or @or @or @or @or @attr 1=36 @attr 4=1 @attr 6=3 @attr 9=32 @attr 2=102 test @attr 1=4 @attr 4=1 @attr 6=3 @attr 9=28 @attr 2=102 test @attr 1=3 6 @attr 4=1 @attr 9=26 @attr 2=102 test @attr 1=4 @attr 4=6 @attr 9=24 @attr 2=102 test @attr 4=6 @attr 5=103 @attr 9=16 @attr 2=102 test @attr 4=6 @attr 5=1 @attr 9=14 @attr 2=102 "test? " @attr 4=6 @attr 9=14 @attr 2=102 test Query: "test" Request: Search biblios ERROR 120 1 1+0 RPN @attrset Bib-1 @or @or @or @or @or @or @attr 1=36 @attr 4=1 @attr 6=3 @attr 9=32 @attr 2=102 " test " @attr 1=4 @attr 4=1 @attr 6=3 @attr 9=28 @attr 2=102 " test " @ attr 1=36 @attr 4=1 @attr 9=26 @attr 2=102 " test " @attr 1=4 @attr 4=6 @attr 9=24 @attr 2=102 " test " @attr 4=6 @attr 5=103 @attr 9=16 @attr 2=102 " test " @attr 4=6 @attr 5=1 @attr 9=14 @attr 2=102 "test? " @attr 4=6 @attr 9=14 @attr 2=102 " test " (Via yaz-client this query yields: [120] Unsupported Truncation attribute -- v2 addinfo '103') -- Actually, it doesn't look like this problem is related to double quotes at all. Interesting... if I disable QueryFuzzy it will work: Query: test Request: Search biblios OK 3 1 1+0 RPN @attrset Bib-1 @or @or @or @or @or @attr 1=36 @attr 4=1 @attr 6=3 @attr 9=32 @attr 2=102 test @attr 1=4 @attr 4=1 @attr 6=3 @attr 9=28 @attr 2=102 test @attr 1=36 @attr 4 =1 @attr 9=26 @attr 2=102 test @attr 1=4 @attr 4=6 @attr 9=24 @attr 2=102 test @attr 4=6 @attr 5=1 @attr 9=14 @attr 2=102 "test? " @attr 4=6 @attr 9=14 @attr 2=102 test Query: "test" Request: Search biblios OK 3 1 1+0 RPN @attrset Bib-1 @or @or @or @or @or @attr 1=36 @attr 4=1 @attr 6=3 @attr 9=32 @attr 2=102 " test " @attr 1=4 @attr 4=1 @attr 6=3 @attr 9=28 @attr 2=102 " test " @attr 1=36 @attr 4=1 @attr 9=26 @attr 2=102 " test " @attr 1=4 @attr 4=6 @attr 9=24 @attr 2=102 " test " @attr 4=6 @attr 5=1 @attr 9=14 @attr 2=102 "test? " @attr 4=6 @attr 9=14 @attr 2=102 " test " -- If I re-enable QueryFuzzy *but* DISABLE QueryStemming it causes catastrophic problems: Query: test Request: Search biblios ERROR 120 1 1+0 RPN @attrset Bib-1 @or @or @or @or @or @attr 1=36 @attr 4=1 @attr 6=3 @attr 9=32 @attr 2=102 test @attr 1=4 @attr 4=1 @attr 6=3 @attr 9=28 @attr 2=102 test @attr 1=36 @a ttr 4=1 @attr 9=26 @attr 2=102 test @attr 1=4 @attr 4=6 @attr 9=24 @attr 2=102 test @attr 4=6 @attr 5=103 @attr 9=16 @attr 2=102 test @attr 4=6 @attr 9=14 @attr 2=102 test Query: "test" Request: Search biblios ERROR 120 1 1+0 RPN @attrset Bib-1 @or @or @or @or @or @attr 1=36 @attr 4=1 @attr 6=3 @attr 9=32 @attr 2=102 " test " @attr 1=4 @attr 4=1 @attr 6=3 @attr 9=28 @attr 2=102 " test " @att r 1=36 @attr 4=1 @attr 9=26 @attr 2=102 " test " @attr 1=4 @attr 4=6 @attr 9=24 @attr 2=102 " test " @attr 4=6 @attr 5=103 @attr 9=16 @attr 2=102 " test " @attr 4=6 @attr 9=14 @attr 2=102 " test " (Via yaz-client this query yield: [120] Unsupported Truncation attribute -- v2 addinfo '103') -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25900 --- Comment #12 from David Cook <dcook@prosentient.com.au> --- (In reply to David Cook from comment #10)
Using koha-testing-docker w/o patch using CHR in /etc/koha/zebradb/etc/default.idx:
Note I have a typo in the QueryAutoTruncate section. By accident, a @attr 2=102 snuck in there. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25900 --- Comment #13 from David Cook <dcook@prosentient.com.au> --- Hmm I've just noticed that I didn't have "UseICU" turned on... but that doesn't look relevant. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25900 --- Comment #14 from David Cook <dcook@prosentient.com.au> --- I'm tempted to mark this as a duplicate of Bug 9715 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25900 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=9715 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25900 --- Comment #15 from Fridolin Somers <fridolin.somers@biblibre.com> --- (In reply to David Cook from comment #13)
Hmm I've just noticed that I didn't have "UseICU" turned on... but that doesn't look relevant.
Indeed, this is only used by XSLT links -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25900 --- Comment #16 from Fridolin Somers <fridolin.somers@biblibre.com> --- (In reply to Fridolin Somers from comment #5)
Oh indeed, even Debian Buster has Zebra 2.0.59.
You can use indexdata repo to upgrade the version : http://ftp.indexdata.dk/debian/
koha-testing-docker uses Debian version no ? This change is harmless I'd say. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25900 --- Comment #17 from David Cook <dcook@prosentient.com.au> --- (In reply to Fridolin Somers from comment #16)
This change is harmless I'd say.
It's harmless but the description is inaccurate. If you renamed this issue to "QueryWeightFields creates unnecessary whitespace", then I would happily test again and probably sign off. But it wouldn't be fixing any functional bug I think. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25900 --- Comment #18 from Fridolin Somers <fridolin.somers@biblibre.com> --- (In reply to David Cook from comment #17)
(In reply to Fridolin Somers from comment #16)
This change is harmless I'd say.
It's harmless but the description is inaccurate. If you renamed this issue to "QueryWeightFields creates unnecessary whitespace", then I would happily test again and probably sign off.
But it wouldn't be fixing any functional bug I think.
Sur, good idea -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25900 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Double quotes breaks search |QueryWeightFields creates |with Zebra and |unnecessary whitespace |QueryWeightFields | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25900 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25900 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #106432|0 |1 is obsolete| | --- Comment #19 from Fridolin Somers <fridolin.somers@biblibre.com> --- Created attachment 114228 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=114228&action=edit Bug 25900: trim spaces in search with Zebra and QueryWeightFields Search with Zebra and system preference QueryWeightFields on builds a query with surrounding double quotes : q=tree => q="tree". Therefore existing double quotes are replaced by a space : q="tree" => q=" tree ". Patch corrects by adding trim on operand after transforming quotes into spaces. Test plan : 1) Use Zebra searchengine and enable system preference QueryWeightFields 2) Go to advanced search 3) Search on two words from a title, surrounded by double quotes. Like "Order of the Phoenix" 4) Check you get same results with an without patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25900 Peter Vashchuk <stalkernoid@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25900 Peter Vashchuk <stalkernoid@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #114228|0 |1 is obsolete| | --- Comment #20 from Peter Vashchuk <stalkernoid@gmail.com> --- Created attachment 121529 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=121529&action=edit Bug 25900: trim spaces in search with Zebra and QueryWeightFields Search with Zebra and system preference QueryWeightFields on builds a query with surrounding double quotes : q=tree => q="tree". Therefore existing double quotes are replaced by a space : q="tree" => q=" tree ". Patch corrects by adding trim on operand after transforming quotes into spaces. Test plan : 1) Use Zebra searchengine and enable system preference QueryWeightFields 2) Go to advanced search 3) Search on two words from a title, surrounded by double quotes. Like "Order of the Phoenix" 4) Check you get same results with an without patch Signed-off-by: Petro Vashchuk <stalkernoid@gmail.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25900 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact| |katrin.fischer@bsz-bw.de -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25900 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25900 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #121529|0 |1 is obsolete| | --- Comment #21 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 121572 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=121572&action=edit Bug 25900: trim spaces in search with Zebra and QueryWeightFields Search with Zebra and system preference QueryWeightFields on builds a query with surrounding double quotes : q=tree => q="tree". Therefore existing double quotes are replaced by a space : q="tree" => q=" tree ". Patch corrects by adding trim on operand after transforming quotes into spaces. Test plan : 1) Use Zebra searchengine and enable system preference QueryWeightFields 2) Go to advanced search 3) Search on two words from a title, surrounded by double quotes. Like "Order of the Phoenix" 4) Check you get same results with an without patch Signed-off-by: Petro Vashchuk <stalkernoid@gmail.com> Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25900 --- Comment #22 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I believe this change makes sense and is harmless. As this is a change to Search.pm we should probably have a unit test verifying it. But I am not sure how hard/easy it would be to test. Leaving final decision to RM. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25900 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Failed QA --- Comment #23 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Please add a unit test! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25900 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|katrin.fischer@bsz-bw.de | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25900 --- Comment #24 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Katrin Fischer from comment #23)
Please add a unit test!
Still waiting for a test - please? -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org