On Fri, May 28, 2010 at 6:07 PM, Koustubha Kale <kmkale@anantcorp.com> wrote:
Hi all, On a test server with latest git clone, I have Devanagari data. I am using zebra with icu. Here is my icu.xml
<icu_chain locale="en_IN.UTF-8"> <transliterate rule="\'>\ "/> <transliterate rule="[:Number:] { '-' > '' "/> <transform rule="[:Control:] Any-Remove"/> <tokenize rule="l"/> <transform rule="[[:WhiteSpace:][:Punctuation:]] Remove"/> <transform rule="NFD"/> <transform rule="[:Nonspacing Mark:] Remove"/> <transform rule="NFC"/> <display/> <casemap rule="l"/> </icu_chain>
Still when I search for any of the following words it seems to make no differentiation between any of the forms of the word.. They all return similar results with sorting order all mixed up. It makes no difference if I have turned off QueryAutoTruncate, QueryFuzzy & QueryStemming.
कांळे काळे काळ काळो काळा काळं काळी काळां काळू काळु
You can also see same behavior on www.granthalaya.org site which is running 3.00.00 with zebra + icu with above icu.xml file.
I have two queries. 1) With all QueryAutoTruncate, QueryFuzzy, QueryStemming etc turned off shouldn't all the above forms of the word be treated as distinct words and NOT appear in other word searches? 2) The words are not sorted at all; even when I select Title A-z or Title Z-a. The ordering changes but its still a incorrect mixed bag.
hdl_laptop had guided me on irc that I need to look into sort-string-utf.chr for the sorting. I read through a lot of stuff on icu and zebra but cant make out how to achieve this. Should I append the correct Devanagari sorting char sequence to the following lines in sort-string-utf.chr?
lowercase {0-9}{a-y}üzæäøöå uppercase {0-9}{A-Y}ÜZÆÄØÖÅ
Please help..
Also I should add that I have tried to remove following lines from icu.xml and rebuilding the index but it makes no difference. <transform rule="[:Control:] Any-Remove"/> <transform rule="[[:WhiteSpace:][:Punctuation:]] Remove"/> <transform rule="NFD"/> <transform rule="[:Nonspacing Mark:] Remove"/> <transform rule="NFC"/> Regards, Koustubha Kale Anant Corporation Contact Details : Address : 103, Armaan Residency, R. W Sawant Road, Nr. Golden Dyes Naka, Thane (w), Maharashtra, India, Pin : 400601. TeleFax : +91-22-21720108, +91-22-21720109 Mobile : +919820715876 Website : http://www.anantcorp.com Blog : http://www.anantcorp.com/blog/?author=2