<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p
{mso-style-priority:99;
margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
p.msochpdefault, li.msochpdefault, div.msochpdefault
{mso-style-name:msochpdefault;
margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
span.emailstyle17
{mso-style-name:emailstyle17;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-AU link="#0563C1" vlink="#954F72"><div class=WordSection1><p class=MsoNormal><a name="_MailEndCompose"><span style='mso-fareast-language:EN-US'>Yikes, that’s not good. That would be great if you could investigate further, and let us know how it goes, and let Indexdata know as well.<o:p></o:p></span></a></p><p class=MsoNormal><span style='mso-fareast-language:EN-US'><o:p> </o:p></span></p><div><p class=MsoNormal>David Cook<o:p></o:p></p><p class=MsoNormal>Systems Librarian<o:p></o:p></p><p class=MsoNormal>Prosentient Systems<o:p></o:p></p><p class=MsoNormal>72/330 Wattle St, Ultimo, NSW 2007<o:p></o:p></p></div><p class=MsoNormal><span style='mso-fareast-language:EN-US'><o:p> </o:p></span></p><div style='border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt'><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span lang=EN-US>From:</span></b><span lang=EN-US> koha-devel-bounces@lists.koha-community.org [mailto:koha-devel-bounces@lists.koha-community.org] <b>On Behalf Of </b>Marcel de Rooy<br><b>Sent:</b> Wednesday, 18 November 2015 6:13 PM<br><b>To:</b> 'Koha-devel' <koha-devel@lists.koha-community.org><br><b>Subject:</b> Re: [Koha-devel] Problematic Zebra Charmaps Equivalences<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><div><p><span style='font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black'>I recently "downgraded" ICU back to CHR in order to overcome Zebra segmentation faults on a complete reindex.<o:p></o:p></span></p><p><span style='font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black'>Should still investigate some further, but have the impression that some Chinese characters made zebraidx crash.<o:p></o:p></span></p><p><span style='font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black'> <o:p></o:p></span></p><div><div class=MsoNormal align=center style='text-align:center'><span style='font-size:12.0pt;font-family:"Times New Roman",serif;color:black'><hr size=2 width="100%" align=center></span></div><div id=divRpF150837><p class=MsoNormal style='margin-bottom:12.0pt'><b><span style='font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black'>Van:</span></b><span style='font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black'> <a href="mailto:koha-devel-bounces@lists.koha-community.org">koha-devel-bounces@lists.koha-community.org</a> [koha-devel-bounces@lists.koha-community.org] namens David Cook [dcook@prosentient.com.au]<br><b>Verzonden:</b> woensdag 18 november 2015 1:25<br><b>Aan:</b> 'Koha-devel'<br><b>Onderwerp:</b> [Koha-devel] Problematic Zebra Charmaps Equivalences</span><span style='font-size:12.0pt;font-family:"Times New Roman",serif;color:black'><o:p></o:p></span></p></div><div><div><p class=MsoNormal><span style='color:black'>Hi all:<o:p></o:p></span></p><p class=MsoNormal><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>Yet another Zebra email from this guy.<o:p></o:p></span></p><p class=MsoNormal><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>I don’t know how many of you are using CHR vs ICU, but CHR is the default for installs, so I’m guessing that it’s quite a few. <o:p></o:p></span></p><p class=MsoNormal><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>Well, there are some issues with how we use the equivalent directive. Hopefully the UTF-8 won’t be stripped out of this message, although I’m guessing it might…<o:p></o:p></span></p><p class=MsoNormal><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>Here’s all instances of the directive in word-phrase-utf.chr:<o:p></o:p></span></p><p class=MsoNormal><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal><span style='color:black'># Characters to be considered equivalent for sorting purposes<o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>equivalent aáàãåâăąȧǎȁȃ<o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>equivalent ӕä(ae)<o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>equivalent ā(aa)<o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>equivalent iíìîịĩĭįǐȉȋ<o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>equivalent ï(ie)<o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>equivalent ī(ii)<o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>equivalent uúùûũŭųűǔȕȗ<o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>equivalent ü(ue)<o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>equivalent ū(uu)<o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>equivalent eéèêẽĕęėěȅȇ<o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>equivalent ëē(ee)<o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>equivalent oóòõôŏǫȯőǒȍȏ<o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>equivalent Œœöø(oe)<o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>equivalent ō(oo)<o:p></o:p></span></p><p class=MsoNormal><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>Firstly, that comment is wrong. “equivalent” isn’t just for sorting purposes. It’s for searching purposes. Indexdata have confirmed that the documentation is wrong about the sorting thing.<o:p></o:p></span></p><p class=MsoNormal><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>So “ie” and ï (if you can’t see this character, it’s the UTF-8 representation of ï) are equivalent. That means searches for “siemon” will get results for “siemon” and “sïmon”. <o:p></o:p></span></p><p class=MsoNormal><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>Now, there is also a “map” directive:<o:p></o:p></span></p><p class=MsoNormal><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>map ï i<o:p></o:p></span></p><p class=MsoNormal><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>This means that “sïmon” is the same as “simon”. Now, “map” affects both indexing and searching. If you have “sïmon” in a record, you can see that it is actually stored as “simon” in Zebra, if you do a search for it and use “format xml” and “elements zebra::index”. <o:p></o:p></span></p><p class=MsoNormal><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>So your search for “siemon” will really get results for “siemon” and “simon”. <o:p></o:p></span></p><p class=MsoNormal><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>This really isn’t ideal. However, you can see why you’d want equivalences. In Scandinavian languages, I think “å” and “aa” are roughly equivalent. They’re spelled differently but they’re the same sound. So if you search for “Gaard”, you might want hits for “Gård” as well. <o:p></o:p></span></p><p class=MsoNormal><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>But you might not want “career” to be equivalent to “carer” as they’re two different words. Or “choose” to be equivalent to “chose”, “sloop” - "slop”, “reef” - "ref”, etc.<o:p></o:p></span></p><p class=MsoNormal><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>--<o:p></o:p></span></p><p class=MsoNormal><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>Unfortunately, I don’t really know what the solution is. For one client, I’ve disabled the equivalent directive where it creates an equivalence between any two letter combination with a one letter combination, as they only have records in English, and it’ll just cause them headaches.<o:p></o:p></span></p><p class=MsoNormal><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>I can see this being useful for multilingual records… although I think many people with multilingual records use ICU. I don’t know ICU well enough to know how it manages characters that English speakers would think of as accents or ligatures. I know you can provide your own normalization with ICU, but I think it does a fair amount on its own as well…<o:p></o:p></span></p><p class=MsoNormal><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>I think some of the difficulties are mentioned here: <a href="http://userguide.icu-project.org/collation/icu-string-search-service" target="_blank">http://userguide.icu-project.org/collation/icu-string-search-service</a>. It also mentions the Danish å/aa example. I don’t know how ICU would know how to handle particular languages… that webpage seems to indicate you can provide a locale to deal with it. <o:p></o:p></span></p><p class=MsoNormal><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>Of course, that doesn’t necessarily solve things. If you have multilingual records with multilingual users, how do you choose your rules? Sure, you might be able to specify a locale at search time (note you can’t do this with Zebra), but what rules did you specify at index time? <o:p></o:p></span></p><p class=MsoNormal><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>As anyone who has watched this video (<a href="https://www.youtube.com/watch?v=0j74jcxSunY" target="_blank">https://www.youtube.com/watch?v=0j74jcxSunY</a>) would know, internationalis(z)ing code has many challenges…<o:p></o:p></span></p><p class=MsoNormal><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>--<o:p></o:p></span></p><p class=MsoNormal><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>Anyway, the reason for this email is mostly just to make you all aware of this issue, and how “equivalent” and “map” work in the Charmap files when using CHR indexing.<o:p></o:p></span></p><p class=MsoNormal><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>Oh, also, if you look at “default.idx”, you’ll see that “sort s” references “charmap sort-string-utf.chr”, but I don’t think sort-string-utf.chr actually exists anywhere…<o:p></o:p></span></p><p class=MsoNormal><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>David Cook<o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>Systems Librarian<o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>Prosentient Systems<o:p></o:p></span></p><p class=MsoNormal><span style='color:black'>72/330 Wattle St, Ultimo, NSW 2007<o:p></o:p></span></p><p class=MsoNormal><span style='color:black'> <o:p></o:p></span></p></div></div></div></div></div></div></body></html>