[Bug 15745] New: C4::Matcher gets CCL parsing error if term contains ? (question mark)
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15745 Bug ID: 15745 Summary: C4::Matcher gets CCL parsing error if term contains ? (question mark) Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: MARC Bibliographic record staging/import Assignee: gmcharlt@gmail.com Reporter: dcook@prosentient.com.au QA Contact: testopia@bugs.koha-community.org I was using C4::Matcher via the record matching rules during an import, and I kept getting the following error: "CCL parsing error (10014) Embedded truncation not supported ZOOM". Query: id-other,st-urx=http://www.skovdenyheter.se?p=38004 I wondered why it was trying to do "embedded truncation" when it shouldn't have been doing any truncation by default... so I thought about wrapping the whole term in double quotation marks. Sure enough, the following worked and didn't produce an error: id-other,st-urx="http://www.skovdenyheter.se?p=38004" You can notice this same problem elsewhere when using CCL. Most noticeably directly with yaz-client: Z> find ti=test? CCL ERROR: Right truncation not supported Z> find ti="test?" Sent searchRequest. Received SearchResponse. Search was a success. Number of hits: 4, setno 2 SearchResult-1: term=test cnt=4 records returned: 0 Elapsed: 0.000816 Also consider the following: Z> find kw,rt=tes* CCL ERROR: Right truncation not supported Z> find kw="tes*" Sent searchRequest. Received SearchResponse. Search was a success. Number of hits: 0, setno 23 SearchResult-1: term=tes cnt=0 records returned: 0 Elapsed: 0.000679 Z> find kw,rt="tes*" Sent searchRequest. Received SearchResponse. Search was a success. Number of hits: 6, setno 24 SearchResult-1: term=tes cnt=6 records returned: 0 Elapsed: 0.001049 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15745 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|gmcharlt@gmail.com |dcook@prosentient.com.au -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15745 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|--- |Small patch Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15745 --- Comment #1 from David Cook <dcook@prosentient.com.au> --- Created attachment 47676 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=47676&action=edit Bug 15745 - C4::Matcher gets CCL parsing error if term contains ? (question mark) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15745 --- Comment #2 from David Cook <dcook@prosentient.com.au> --- I'll write a better description and test plan next week when I have more time... -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15745 --- Comment #3 from M. Tompsett <mtompset@hotmail.com> --- Comment on attachment 47676 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=47676 Bug 15745 - C4::Matcher gets CCL parsing error if term contains ? (question mark) Review of attachment 47676: --> (https://bugs.koha-community.org/bugzilla3/page.cgi?id=splinter.html&bug=15745&attachment=47676) ----------------------------------------------------------------- ::: C4/Matcher.pm @@ +658,5 @@
else { my $phr = C4::Context->preference('AggressiveMatchOnISBN') ? ',phr' : q{}; $query = join( " or ", + map { "$matchpoint->{'index'}$phr=\"$_\"" } @source_keys ); + #NOTE: double-quote the values so you don't get a "Embedded truncation not supported" error when a term has a ? in it.
What about the test case where you want to search for double quotes? Escaping is needed. Perhaps there is another way? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15745 --- Comment #4 from David Cook <dcook@prosentient.com.au> --- (In reply to M. Tompsett from comment #3)
What about the test case where you want to search for double quotes? Escaping is needed. Perhaps there is another way?
I disagree. Why do you think escaping is needed? Can you point me to the particular test case or query in mind? Firstly, the values are coming from the MARC record, so I doubt there will be many double quotes in that data. Secondly, even if there are double quotes in the data, CCL2RPN takes them into account. We can see what happens if we double up our double quotes on the front/back of the query: Z> find id-other,st-urx="http://libris.kb.se/resource/bib/219553" Sent searchRequest. Received SearchResponse. Search was a success. Number of hits: 11, setno 31 SearchResult-1: term=http://libris.kb.se/resource/bib/219553 cnt=11 records returned: 0 Elapsed: 0.000886 BECOMES: @attrset Bib-1 @attr 1=9012 @attr 4=104 http://libris.kb.se/resource/bib/219553 Z> find id-other,st-urx=""http://libris.kb.se/resource/bib/219553"" Sent searchRequest. Received SearchResponse. Search was a success. Number of hits: 11, setno 32 SearchResult-1: term=http://libris.kb.se/resource/bib/219553 cnt=11 records returned: 0 Elapsed: 0.000792 BECOMES: @attrset Bib-1 @attr 1=9012 @attr 4=104 http://libris.kb.se/resource/bib/219553 Z> f ti,phr=""This is a test"" Sent searchRequest. Received SearchResponse. Search was a success. Number of hits: 4, setno 33 SearchResult-1: term=This cnt=4, term=is cnt=4, term=a cnt=4, term=test cnt=4 records returned: 0 Elapsed: 0.001079 BECOMES: @attrset Bib-1 @attr 1=4 @attr 4=1 "This is a test" Z> f ti,phr="This is a test" Sent searchRequest. Received SearchResponse. Search was a success. Number of hits: 4, setno 34 SearchResult-1: term=This cnt=4, term=is cnt=4, term=a cnt=4, term=test cnt=4 records returned: 0 Elapsed: 0.000961 BECOMES: @attrset Bib-1 @attr 1=4 @attr 4=1 "This is a test" -- I'm guessing you're referring to a case where the data contains a double quote and you're searching to match a double quote... but Zebra should strip out double quotes during indexing in most cases. It might not for URLs, but RFC3986 stipulates that you should be percent encoding double quotes as %22 anyway so that shouldn't be an issue. I'll look at an example of that in a moment... -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15745 --- Comment #5 from David Cook <dcook@prosentient.com.au> --- Interestingly enough, Zebra doesn't like it when you embed double quotes in a URL. http://prosentient.com.au?test="test" Becomes the following during indexing: http://prosentient.com.au?test=@test@ And the only way to retrieve it is to use the @ symbol instead of the double quotes in the query as well. Z> find uri,st-urx=http://prosentient.com.au?test="test" CCL ERROR: Embedded truncation not supported Z> find uri,st-urx="http://prosentient.com.au?test="test"" Sent searchRequest. Received SearchResponse. Search was a success. Number of hits: 0, setno 11 SearchResult-1: term=http://prosentient.com.au?test=test cnt=0 records returned: 0 Elapsed: 0.000750 Z> find uri,st-urx="http://prosentient.com.au?test=@test@" Sent searchRequest. Received SearchResponse. Search was a success. Number of hits: 1, setno 10 SearchResult-1: term=http://prosentient.com.au?test=@test@ cnt=1 records returned: 0 Elapsed: 0.000918 Now I'll try an example with normal word and phrase indexes... -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15745 --- Comment #6 from David Cook <dcook@prosentient.com.au> --- As expected, the double quote is removed during normalization during indexing and retrieval for the phrase index: 245 00 $a This is a "test" / $c by David Cook Z> find ti,phr="This is a "test"" Sent searchRequest. Received SearchResponse. Search was a success. Number of hits: 4, setno 4 SearchResult-1: term=This cnt=4, term=is cnt=4, term=a cnt=8, term=test cnt=4 records returned: 0 Elapsed: 0.001117 Z> show 1 245 00 $a This is a "test" / $c by David Cook Z> find ti,phr="This is a test" Sent searchRequest. Received SearchResponse. Search was a success. Number of hits: 4, setno 5 SearchResult-1: term=This cnt=4, term=is cnt=4, term=a cnt=8, term=test cnt=4 records returned: 0 Elapsed: 0.000991 Z> show 1 245 00 $a This is a "test" / $c by David Cook -- We can see this using "elements zebra::index" too: <index name="Title" type="p" seq="112">this is a test by david cook</index> We can see it with the word register too: <index name="Title" type="w" seq="113">this</index> <index name="Title" type="w" seq="114">is</index> <index name="Title" type="w" seq="115">a</index> <index name="Title" type="w" seq="116">test</index> <index name="Title" type="w" seq="117">by</index> <index name="Title" type="w" seq="118">david</index> <index name="Title" type="w" seq="119">cook</index> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15745 --- Comment #7 from David Cook <dcook@prosentient.com.au> --- In other words, I think my patch still holds up. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15745 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |10662 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 [Bug 10662] Build OAI-PMH Harvesting Client -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15745 Olli-Antti Kivilahti <olli-antti.kivilahti@jns.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #47676|0 |1 is obsolete| | --- Comment #8 from Olli-Antti Kivilahti <olli-antti.kivilahti@jns.fi> --- Created attachment 49255 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=49255&action=edit Bug 15745 - C4::Matcher gets CCL parsing error if term contains ? (question mark) Signed-off-by: Olli-Antti Kivilahti <olli-antti.kivilahti@jns.fi> Also fixes ! and + Rebased to master -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15745 Olli-Antti Kivilahti <olli-antti.kivilahti@jns.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off CC| |olli-antti.kivilahti@jns.fi -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15745 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15745 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #49255|0 |1 is obsolete| | --- Comment #9 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 49809 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=49809&action=edit Bug 15745: C4::Matcher gets CCL parsing error if term contains ? (question mark) Signed-off-by: Olli-Antti Kivilahti <olli-antti.kivilahti@jns.fi> Also fixes ! and + Rebased to master Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar> It makes perfect sense and works as expected. This part of the code is too under-tested so no point requiring a regression test for such a simple change. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15745 Brendan Gallagher <brendan@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |brendan@bywatersolutions.co | |m Status|Passed QA |Pushed to Master --- Comment #10 from Brendan Gallagher <brendan@bywatersolutions.com> --- Pushed to Master - Should be in the May 2016 release. Thanks! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15745 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to Master |Pushed to Stable CC| |julian.maurice@biblibre.com --- Comment #11 from Julian Maurice <julian.maurice@biblibre.com> --- Patch pushed to 3.22.x, will be in 3.22.6 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15745 Frédéric Demians <frederic@tamil.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |frederic@tamil.fr --- Comment #12 from Frédéric Demians <frederic@tamil.fr> --- Pushed to 3.20.x, will be in 3.20.11. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15745 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks|10662 | Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 [Bug 10662] Build OAI-PMH Harvesting Client -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15745 Patrick Robitaille <patrick.robitaille@ccsr.qc.ca> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |patrick.robitaille@ccsr.qc. | |ca -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org