[Bug 27581] New: UseICU is a misnamed systempreference
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27581 Bug ID: 27581 Summary: UseICU is a misnamed systempreference Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Architecture, internals, and plumbing Assignee: koha-bugs@lists.koha-community.org Reporter: nick@bywatersolutions.com QA Contact: testopia@bugs.koha-community.org This system preference has no effect on whether ICU is used or not, it simply switches the character used to quote subjects for searching. I am not really sure whether this pref is needed, or why other search indexes don't seem to be affected by it. While we should probably look towards removing it, the first step is to aptly describe it -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27581 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff Patch complexity|--- |Trivial patch -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27581 --- Comment #1 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 116078 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=116078&action=edit Bug 27581: Rename USeICU preference to UseICUStyleQuotes Rather thna have a poorly named preference with an explanation of how it doesn't do what it implies it does, let's rename it! To test: 1 - Set UseICU to 'Using' 2 - Go to the details page of a record in the staff interface 3 - Hover over a subject heading that has subfields 4 - Note the link has curly brackets around the subject, like: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=su:{Winnie-the-Pooh Fictitious character} 5 - Change UseICU to 'Not using' 6 - Note the link is now: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Winnie-the-Pooh Fictitious character" 7 - Repeat on OPAC 8 - Apply patch 9 - Restart all 10 - Repeat tests, the behaviour has not changed 11 - Read the new syspref description and confirm it makes sense -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27581 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au, | |katrin.fischer@bsz-bw.de Assignee|koha-bugs@lists.koha-commun |nick@bywatersolutions.com |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27581 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27581 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #116078|0 |1 is obsolete| | --- Comment #2 from Andrew Fuerste-Henry <andrew@bywatersolutions.com> --- Created attachment 116082 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=116082&action=edit Bug 27581: Rename USeICU preference to UseICUStyleQuotes Rather thna have a poorly named preference with an explanation of how it doesn't do what it implies it does, let's rename it! To test: 1 - Set UseICU to 'Using' 2 - Go to the details page of a record in the staff interface 3 - Hover over a subject heading that has subfields 4 - Note the link has curly brackets around the subject, like: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=su:{Winnie-the-Pooh Fictitious character} 5 - Change UseICU to 'Not using' 6 - Note the link is now: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Winnie-the-Pooh Fictitious character" 7 - Repeat on OPAC 8 - Apply patch 9 - Restart all 10 - Repeat tests, the behaviour has not changed 11 - Read the new syspref description and confirm it makes sense Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27581 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Failed QA --- Comment #3 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Hi Nick, I am sorry, but some unrelated changes slipped into this patch: koha-tmpl/intranet-tmpl/prog/en/modules/tags/review.tt -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27581 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27581 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #116082|0 |1 is obsolete| | --- Comment #4 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 116154 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=116154&action=edit Bug 27581: Rename USeICU preference to UseICUStyleQuotes Rather thna have a poorly named preference with an explanation of how it doesn't do what it implies it does, let's rename it! To test: 1 - Set UseICU to 'Using' 2 - Go to the details page of a record in the staff interface 3 - Hover over a subject heading that has subfields 4 - Note the link has curly brackets around the subject, like: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=su:{Winnie-the-Pooh Fictitious character} 5 - Change UseICU to 'Not using' 6 - Note the link is now: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Winnie-the-Pooh Fictitious character" 7 - Repeat on OPAC 8 - Apply patch 9 - Restart all 10 - Repeat tests, the behaviour has not changed 11 - Read the new syspref description and confirm it makes sense Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27581 --- Comment #5 from Nick Clemens <nick@bywatersolutions.com> --- (In reply to Katrin Fischer from comment #3)
Hi Nick, I am sorry, but some unrelated changes slipped into this patch: koha-tmpl/intranet-tmpl/prog/en/modules/tags/review.tt
Oops, reverted those -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27581 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |kyle@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27581 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |Manual -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27581 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27581 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #116154|0 |1 is obsolete| | --- Comment #6 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 116455 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=116455&action=edit Bug 27581: Rename USeICU preference to UseICUStyleQuotes Rather thna have a poorly named preference with an explanation of how it doesn't do what it implies it does, let's rename it! To test: 1 - Set UseICU to 'Using' 2 - Go to the details page of a record in the staff interface 3 - Hover over a subject heading that has subfields 4 - Note the link has curly brackets around the subject, like: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=su:{Winnie-the-Pooh Fictitious character} 5 - Change UseICU to 'Not using' 6 - Note the link is now: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Winnie-the-Pooh Fictitious character" 7 - Repeat on OPAC 8 - Apply patch 9 - Restart all 10 - Repeat tests, the behaviour has not changed 11 - Read the new syspref description and confirm it makes sense Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com> Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27581 --- Comment #7 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 116456 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=116456&action=edit Bug 27581: (QA follow-up) Fix occurence in it-IT custom.sql -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27581 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|UseICU is a misnamed |Rename UseICU system |systempreference |preference to | |UseICUStyleQuotes -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27581 --- Comment #8 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 116498 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=116498&action=edit Bug 27581: Fix missing occurrences in tests -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27581 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to master Version(s)| |21.05.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27581 --- Comment #9 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Pushed to master for 21.05, thanks to everybody involved! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27581 --- Comment #10 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Sorry, not sure how the test ones escaped me. hmpf. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27581 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m --- Comment #11 from Fridolin Somers <fridolin.somers@biblibre.com> --- I prefer not change syspref on stable branch. This may be used in home-made XSLT files. Not pushed to 20.11.x -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27581 Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |caroline.cyr-la-rose@inlibr | |o.com --- Comment #12 from Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com> --- Just putting here what the manual said about this pref. ICU is a set of code libraries providing Unicode and Globalization support for software applications. What this means is ICU Zebra indexing is only necessary if you use non-roman characters in your cataloging. If using ICU Zebra indexing you will want to not use QueryFuzzy. This setting will not affect Zebra indexing, it should only be used to tell Koha that you have activated ICU indexing if you have actually done so, since there is no way for Koha to figure this out on its own. Does the 'new' system preference still block QueryFuzzy from working? Do the different quotes (either double quotes or curly brackets) have an influence when tracing subjects using non-roman characters? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27581 --- Comment #13 from Nick Clemens <nick@bywatersolutions.com> --- (In reply to Caroline Cyr La Rose from comment #12)
Just putting here what the manual said about this pref.
ICU is a set of code libraries providing Unicode and Globalization support for software applications. What this means is ICU Zebra indexing is only necessary if you use non-roman characters in your cataloging. If using ICU Zebra indexing you will want to not use QueryFuzzy.
This setting will not affect Zebra indexing, it should only be used to tell Koha that you have activated ICU indexing if you have actually done so, since there is no way for Koha to figure this out on its own.
Does the 'new' system preference still block QueryFuzzy from working? Do the different quotes (either double quotes or curly brackets) have an influence when tracing subjects using non-roman characters?
There are several places on the wiki that cover using ICU chains - it is much more involved than a syspref swtich (well, it's a switch in two config files) https://wiki.koha-community.org/wiki/ICU_chains_configuration https://wiki.koha-community.org/wiki/ICU_do_not_undiacritic https://wiki.koha-community.org/wiki/Correcting_Search_of_Arabic_records ICU still doesn't work with queryfuzzy afaik I did not find the documentation from Zebra on the use of quotes, but I assume ICU requires the change - many of our partner sites were using non-ICU indexing with ICU style quotes, so at least one=-way it is not harmful -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27581 --- Comment #14 from David Cook <dcook@prosentient.com.au> --- (In reply to Nick Clemens from comment #13)
I did not find the documentation from Zebra on the use of quotes, but I assume ICU requires the change - many of our partner sites were using non-ICU indexing with ICU style quotes, so at least one=-way it is not harmful
Actually, now that I think about this again, I think that Koha's idea of "ICU quotes" is totally wrong. Koha doesn't speak ICU or CHR to Zebra. Koha just sends PQF queries to Zebra (and it uses a CCL2RPN conversion to produce these PQF queries). The quotes are relevant only in terms of CCL and PQF queries. So... they're relevant regardless of indexing mode (which basically just governs normalization and tokenization). Now... Bug 27299 shows and solves a tokenization issue with ICU and phrase registers, and that caused there to be a difference in how records were being indexed. I see that there is a TraceCompleteSubfields system preference which adds a qualifier of ",complete-subfield", which would cause the phrase register to be used as it's a 6=2 attribute (see https://github.com/indexdata/idzebra/blob/master/tab/default.idx for details on the register) Technically speaking, we should always be wrapping the operands in double quotes so that queries are like this: au="Cook, David" and su="ICU". -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27581 --- Comment #15 from David Cook <dcook@prosentient.com.au> --- Oh I see... with UseICU, we're using curly braces instead of double quotes. Yeah... that's not an ICU thing. You can see some discussion about this here: http://lists.indexdata.dk/pipermail/zebralist/2007-July/001674.html http://lists.indexdata.dk/pipermail/yazlist/2001-September/000090.html I think it might be a case of poor string handling in Perl in the past. Going to dredge up some more detail.. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27581 --- Comment #16 from David Cook <dcook@prosentient.com.au> --- I think that curly braces are just an undocumented synonym for double quotes. We could reach out to Indexdata for comment on that. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27581 --- Comment #17 from David Cook <dcook@prosentient.com.au> --- This is using ICU: Z> find su,complete-subfield={Rock musicians United States Biography 1705} Sent searchRequest. Received SearchResponse. Search was a success. Number of hits: 2, setno 16 SearchResult-1: term=Rock musicians United States Biography 1705 cnt=2 records returned: 0 Elapsed: 0.000290 Z> find su,complete-subfield="Rock musicians United States Biography 1705" Sent searchRequest. Received SearchResponse. Search was a success. Number of hits: 2, setno 17 SearchResult-1: term=Rock musicians United States Biography 1705 cnt=2 records returned: 0 Elapsed: 0.000361 So UseICUStyleQuotes doesn't really make sense to me. It's probably just hiding a bug in Koha. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27581 --- Comment #18 from David Cook <dcook@prosentient.com.au> --- Note that the above used the fix from Bug 27299, but I just tried again without the tokenization fix, and both {} and "" works fine. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27581 --- Comment #19 from David Cook <dcook@prosentient.com.au> --- Looking at Bug 7092 which introduced UseICU... it looks like this was always a workaround. If someone has a test plan that shows ICU not working with "" or CHR not working with {}, I'd happily test it. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27581 Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|Manual | -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org