[Bug 35455] New: ICU does not strip = when indexing/searching
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35455 Bug ID: 35455 Summary: ICU does not strip = when indexing/searching Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Searching - Zebra Assignee: koha-bugs@lists.koha-community.org Reporter: dcook@prosentient.com.au The CHR config file /etc/koha/zebradb/etc/word-phrase-utf.chr replaces = with a space, but the ICU config files /etc/koha/zebradb/etc/words-icu.xml and /etc/koha/zebradb/etc/phrases-icu.xml do not transform the = character. I encountered this problem because the "Show analytics" link wasn't appearing for a journal record, but it will affect other search contexts too. Note on https://unicode-org.github.io/icu/userguide/strings/regexp.html it says: "The property expression \p{punct} differs in what it matches. Java matches matches any of !"#$%&'()*+,-./:;<=>?@[\]^_`{|}~. From that list, ICU omits $+<=>^`|~ ICU follows the recommendations from Unicode UTS-1" Arguably we should remove <=>^`|~ during ICU normalization but I'm just focusing on = for now, as the colon : gets changed to = during searches, and subtitles frequently are introduced by : in English. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35455 --- Comment #1 from David Cook <dcook@prosentient.com.au> --- For example: Record 1: 245$a Awesome title: awesome subtitle Record 2: 245$a Cool article 773$t Awesome title: awesome subtitle -- At indexing time, this will be indexed as "Awesome title awesome subtitle" as the ICU removes the punctuation. At search time, we can't use "Awesome title= awesome subtitle" because the tokens "title" and "title=" won't match. We've got to remove the = since at indexing time it was a : -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35455 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35455 --- Comment #2 from David Cook <dcook@prosentient.com.au> --- Created attachment 159422 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=159422&action=edit Bug 35455: Remove = in ICU for indexing/searching This change more closely aligns ICU and CHR so that ICU also removes the = character. This fixes issues in ICU when searching with a : which gets transformed into a =. Without this change, the Analytics features won't work for titles with a colon in them. Test plan: 0. Apply the patch and import bibs from Bugzilla (using Staged MARC tools) 1. cp ./etc/zebradb/etc/phrases-icu.xml /etc/koha/zebradb/etc/phrases-icu.xml 2. cp ./etc/zebradb/etc/words-icu.xml /etc/koha/zebradb/etc/words-icu.xml 3. vi /etc/koha/zebradb/etc/default.idx Change "charmap word-phrase-utf.chr" to "icuchain words-icu.xml" for "index w" and "icuchain phrases-icu.xml" for "index p" 4. koha-zebra --stop kohadev 5. pkill zebrasrv 6. koha-zebra --start kohadev 7. koha-rebuild-zebra -a -b -f -v kohadev 8. Search for "Awesome title" and open the detail page 9. Note that the "Analytics: Show analytics" line shows up 10. Click that link 11. Note that it opens the "Cool article" record and it displays "In: Awesome title: awesome subtitle" 12. Click that link 13. Note that it opens the "Awesome title" record -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35455 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the| |This change fixes an issue release notes| |with Zebra ICU searching | |where titles with colons | |aren't properly searchable, | |especially when used with | |Analytics. | | | |A full re-index | |of Zebra is needed for this | |change to take effect. Assignee|koha-bugs@lists.koha-commun |dcook@prosentient.com.au |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35455 --- Comment #3 from David Cook <dcook@prosentient.com.au> --- Created attachment 159423 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=159423&action=edit Record 1 Record 1 for testing -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35455 --- Comment #4 from David Cook <dcook@prosentient.com.au> --- Created attachment 159424 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=159424&action=edit Record 2 for testing Record 2 for testing -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35455 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35455 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #159422|0 |1 is obsolete| | --- Comment #5 from David Nind <david@davidnind.com> --- Created attachment 159512 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=159512&action=edit Bug 35455: Remove = in ICU for indexing/searching This change more closely aligns ICU and CHR so that ICU also removes the = character. This fixes issues in ICU when searching with a : which gets transformed into a =. Without this change, the Analytics features won't work for titles with a colon in them. Test plan: 0. Apply the patch and import bibs from Bugzilla (using Staged MARC tools) 1. cp ./etc/zebradb/etc/phrases-icu.xml /etc/koha/zebradb/etc/phrases-icu.xml 2. cp ./etc/zebradb/etc/words-icu.xml /etc/koha/zebradb/etc/words-icu.xml 3. vi /etc/koha/zebradb/etc/default.idx Change "charmap word-phrase-utf.chr" to "icuchain words-icu.xml" for "index w" and "icuchain phrases-icu.xml" for "index p" 4. koha-zebra --stop kohadev 5. pkill zebrasrv 6. koha-zebra --start kohadev 7. koha-rebuild-zebra -a -b -f -v kohadev 8. Search for "Awesome title" and open the detail page 9. Note that the "Analytics: Show analytics" line shows up 10. Click that link 11. Note that it opens the "Cool article" record and it displays "In: Awesome title: awesome subtitle" 12. Click that link 13. Note that it opens the "Awesome title" record Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35455 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |martin.renvoize@ptfs-europe | |.com QA Contact| |martin.renvoize@ptfs-europe | |.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35455 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #159512|0 |1 is obsolete| | --- Comment #6 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 160011 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=160011&action=edit Bug 35455: Remove = in ICU for indexing/searching This change more closely aligns ICU and CHR so that ICU also removes the = character. This fixes issues in ICU when searching with a : which gets transformed into a =. Without this change, the Analytics features won't work for titles with a colon in them. Test plan: 0. Apply the patch and import bibs from Bugzilla (using Staged MARC tools) 1. cp ./etc/zebradb/etc/phrases-icu.xml /etc/koha/zebradb/etc/phrases-icu.xml 2. cp ./etc/zebradb/etc/words-icu.xml /etc/koha/zebradb/etc/words-icu.xml 3. vi /etc/koha/zebradb/etc/default.idx Change "charmap word-phrase-utf.chr" to "icuchain words-icu.xml" for "index w" and "icuchain phrases-icu.xml" for "index p" 4. koha-zebra --stop kohadev 5. pkill zebrasrv 6. koha-zebra --start kohadev 7. koha-rebuild-zebra -a -b -f -v kohadev 8. Search for "Awesome title" and open the detail page 9. Note that the "Analytics: Show analytics" line shows up 10. Click that link 11. Note that it opens the "Cool article" record and it displays "In: Awesome title: awesome subtitle" 12. Click that link 13. Note that it opens the "Awesome title" record Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35455 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA --- Comment #7 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- All works as described and makes sense. Passing QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35455 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to master Version(s)| |24.05.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35455 --- Comment #8 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Pushed for 24.05! Well done everyone, thank you! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35455 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m Status|Pushed to master |Pushed to stable Version(s)|24.05.00 |24.05.00,23.11.02 released in| | --- Comment #9 from Fridolin Somers <fridolin.somers@biblibre.com> --- Pushed to 23.11.x for 23.11.02 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35455 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|24.05.00,23.11.02 |24.05.00,23.11.02,23.05.08 released in| | CC| |lucas@bywatersolutions.com Status|Pushed to stable |Pushed to oldstable --- Comment #10 from Lucas Gass <lucas@bywatersolutions.com> --- Backported to 23.05.x for upcoming 23.05.08 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35455 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC|martin.renvoize@ptfs-europe | |.com | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35455 Wainui Witika-Park <wainuiwitikapark@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to oldstable |RESOLVED CC| |wainuiwitikapark@catalyst.n | |et.nz Resolution|--- |FIXED --- Comment #11 from Wainui Witika-Park <wainuiwitikapark@catalyst.net.nz> --- Not backporting to 22.11 -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org