proper sorting order and search for Swedish characters and ISBN search
Hello, I have been trying several things to try to sort this out, and haven't found a way (yet) that would avoid ugly compromises. The issue is: in Swedish ä, ö and å are separate letters. Not variants of a and o. This means that searching for å shouldn't bring up a. When sorting, they belong to the very end of the alphabet, after z, not along a and o. ICU has some kind of setting that allows to standardize isbns for searching, so you can search for any hyphenated variant, and still find the isbn that actually appears in your data. I *think* this works by removing all the hyphens in the index, but i am not quite sure. Anyway, in order to get the Swedish letters right, we had to do some specific chr configuration. Doing this forces us not to use ICU, which means giving up on standardized isbn search. Alternatives are transforming all the isbns in the records (quite uly). Or using the SearchWithISBNVariations syspref, but as far as i understand, this only works when using the dedicated isbn index, not in the simple search. Has anyone found a more satisfactory way of handling this? Thanks, -- Gaetan Boisson Chef de projet bibliothécaire BibLibre +33(0)6 52 42 51 29 108 rue Breteuil 13006 Marseille gaetan.boisson@biblibre.com
Hi Gaetan and all, Il 04/11/2016 17:20, Gaetan Boisson ha scritto:
The issue is: in Swedish ä, ö and å are separate letters. Not variants of a and o. This means that searching for å shouldn't bring up a. When sorting, they belong to the very end of the alphabet, after z, not along a and o.
ICU has some kind of setting that allows to standardize isbns for searching, so you can search for any hyphenated variant, and still find the isbn that actually appears in your data. I *think* this works by removing all the hyphens in the index, but i am not quite sure.
Anyway, in order to get the Swedish letters right, we had to do some specific chr configuration. Doing this forces us not to use ICU, which means giving up on standardized isbn search.
I think you speaking about general index. zebradb/lang_defs/en/sort-string-utf.chr 7 lowercase {0-9}{a-y}üzæäøöå 8 uppercase {0-9}{A-Y}ÜZÆÄØÖÅ [..] 16 # equivalent æä(ae) 17 # equivalent øö(oe) 18 # equivalent å(aa) 19 # equivalent uü zebradb/etc/words-icu.xml 2 <transliterate rule="{ œ > oe "/> 3 <transliterate rule="{ Œ > oe "/> 4 <transliterate rule="{ æ > ae "/> 5 <transliterate rule="{ Æ > ae "/> zebradb/etc/phrases-icu.xml 2 <transliterate rule="{ œ > oe "/> 3 <transliterate rule="{ Œ > oe "/> 4 <transliterate rule="{ æ > ae "/> 5 <transliterate rule="{ Æ > ae "/> Is it not enough to change only those configurations to fix your problem ? Which configuration do force you to CHR conf instead of ICU ? Bye Zeno Tajoli -- Zeno Tajoli /SVILUPPO PRODOTTI CINECA/ - Automazione Biblioteche Email: z.tajoli@cineca.it Fax: 051/6132198 *CINECA* Consorzio Interuniversitario - Sede operativa di Segrate (MI)
On Fri, Nov 04, 2016 at 05:20:03PM +0100, Gaetan Boisson wrote:
Hello,
I have been trying several things to try to sort this out, and haven't found a way (yet) that would avoid ugly compromises.
The issue is: in Swedish ä, ö and å are separate letters. Not variants of a and o. This means that searching for å shouldn't bring up a. When sorting, they belong to the very end of the alphabet, after z, not along a and o.
In ICU, setting the locale parameter to "sv" (swedish) should do this. I cant seem to get this to work in yaz-icu but there are tests in the yaz code base for passing locale to strings. However I'm not sure how to correctly specify this for yaz the documentation is way too sketchy. Colin -- Colin Campbell Chief Software Engineer, PTFS Europe Limited Content Management and Library Solutions +44 (0) 800 756 6803 (phone) +44 (0) 7759 633626 (mobile) colin.campbell@ptfs-europe.com skype: colin_campbell2 http://www.ptfs-europe.com
-----Original Message----- From: koha-devel-bounces@lists.koha-community.org [mailto:koha-devel- bounces@lists.koha-community.org] On Behalf Of Colin Campbell Sent: Friday, 25 November 2016 3:32 AM To: koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] proper sorting order and search for Swedish characters and ISBN search
On Fri, Nov 04, 2016 at 05:20:03PM +0100, Gaetan Boisson wrote:
Hello,
I have been trying several things to try to sort this out, and haven't found a way (yet) that would avoid ugly compromises.
The issue is: in Swedish ä, ö and å are separate letters. Not variants of a and o. This means that searching for å shouldn't bring up a. When sorting, they belong to the very end of the alphabet, after z, not along a and o.
In ICU, setting the locale parameter to "sv" (swedish) should do this. I cant seem to get this to work in yaz-icu but there are tests in the yaz code
passing locale to strings. However I'm not sure how to correctly specify
Considering the folk at IndexData are Danish, perhaps they could provide some insight more directly. David Cook Systems Librarian Prosentient Systems 72/330 Wattle St Ultimo, NSW 2007 Australia Office: 02 9212 0899 Direct: 02 8005 0595 base for this for
yaz the documentation is way too sketchy.
Colin
-- Colin Campbell Chief Software Engineer, PTFS Europe Limited Content Management and Library Solutions +44 (0) 800 756 6803 (phone) +44 (0) 7759 633626 (mobile) colin.campbell@ptfs-europe.com skype: colin_campbell2
http://www.ptfs-europe.com _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
Digging this back up as we finally found a solution. And by "we" i mean Fridolin Somers, whose knowledge of ICU, zebra and indexing never ceases to amaze me! Setting the ICU locale to sv-SE was not enough it seems, because this still transliterated å, ä, ö, Å, Ä and Ö to a, o, A and O. We just had to add a rule to exclude them from the transliteration: <transform rule="[^åäöÅÄÖ] NFD"/><!-- do not undiactric some characters --> Meh, sounds easy enough now! Hopefully this will come in handy to someone. Le 12/01/2017 à 02:51, David Cook a écrit :
Considering the folk at IndexData are Danish, perhaps they could provide some insight more directly.
David Cook Systems Librarian Prosentient Systems 72/330 Wattle St Ultimo, NSW 2007 Australia
Office: 02 9212 0899 Direct: 02 8005 0595
-----Original Message----- From: koha-devel-bounces@lists.koha-community.org [mailto:koha-devel- bounces@lists.koha-community.org] On Behalf Of Colin Campbell Sent: Friday, 25 November 2016 3:32 AM To: koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] proper sorting order and search for Swedish characters and ISBN search
Hello,
I have been trying several things to try to sort this out, and haven't found a way (yet) that would avoid ugly compromises.
The issue is: in Swedish ä, ö and å are separate letters. Not variants of a and o. This means that searching for å shouldn't bring up a. When sorting, they belong to the very end of the alphabet, after z, not along a and o. In ICU, setting the locale parameter to "sv" (swedish) should do this. I cant seem to get this to work in yaz-icu but there are tests in the yaz code
On Fri, Nov 04, 2016 at 05:20:03PM +0100, Gaetan Boisson wrote: base for passing locale to strings. However I'm not sure how to correctly specify this for yaz the documentation is way too sketchy.
Colin
-- Colin Campbell Chief Software Engineer, PTFS Europe Limited Content Management and Library Solutions +44 (0) 800 756 6803 (phone) +44 (0) 7759 633626 (mobile) colin.campbell@ptfs-europe.com skype: colin_campbell2
http://www.ptfs-europe.com _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Gaetan Boisson Chef de projet bibliothécaire BibLibre +33(0)6 52 42 51 29 108 rue Breteuil 13006 Marseille gaetan.boisson@biblibre.com
Thx for sharing! On 01.02.2017 15:18, Gaetan Boisson wrote:
Digging this back up as we finally found a solution. And by "we" i mean Fridolin Somers, whose knowledge of ICU, zebra and indexing never ceases to amaze me!
Setting the ICU locale to sv-SE was not enough it seems, because this still transliterated å, ä, ö, Å, Ä and Ö to a, o, A and O.
We just had to add a rule to exclude them from the transliteration:
<transform rule="[^åäöÅÄÖ] NFD"/><!-- do not undiactric some characters -->
Meh, sounds easy enough now!
Hopefully this will come in handy to someone.
Le 12/01/2017 à 02:51, David Cook a écrit :
Considering the folk at IndexData are Danish, perhaps they could provide some insight more directly.
David Cook Systems Librarian Prosentient Systems 72/330 Wattle St Ultimo, NSW 2007 Australia
Office: 02 9212 0899 Direct: 02 8005 0595
-----Original Message----- From: koha-devel-bounces@lists.koha-community.org [mailto:koha-devel- bounces@lists.koha-community.org] On Behalf Of Colin Campbell Sent: Friday, 25 November 2016 3:32 AM To: koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] proper sorting order and search for Swedish characters and ISBN search
Hello,
I have been trying several things to try to sort this out, and haven't found a way (yet) that would avoid ugly compromises.
The issue is: in Swedish ä, ö and å are separate letters. Not variants of a and o. This means that searching for å shouldn't bring up a. When sorting, they belong to the very end of the alphabet, after z, not along a and o. In ICU, setting the locale parameter to "sv" (swedish) should do
On Fri, Nov 04, 2016 at 05:20:03PM +0100, Gaetan Boisson wrote: this. I cant seem to get this to work in yaz-icu but there are tests in the yaz code base for passing locale to strings. However I'm not sure how to correctly specify this for yaz the documentation is way too sketchy.
Colin
-- Colin Campbell Chief Software Engineer, PTFS Europe Limited Content Management and Library Solutions +44 (0) 800 756 6803 (phone) +44 (0) 7759 633626 (mobile) colin.campbell@ptfs-europe.com skype: colin_campbell2
http://www.ptfs-europe.com _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
Gaetan, I've added your entry here: https://wiki.koha-community.org/wiki/ICU_Chains_Library#Swedish Please let me know if I've made any copy/paste errors. Thanks, --Barton On Wed, Feb 1, 2017 at 9:18 AM, Gaetan Boisson <gaetan.boisson@biblibre.com> wrote:
Digging this back up as we finally found a solution. And by "we" i mean Fridolin Somers, whose knowledge of ICU, zebra and indexing never ceases to amaze me!
Setting the ICU locale to sv-SE was not enough it seems, because this still transliterated å, ä, ö, Å, Ä and Ö to a, o, A and O.
We just had to add a rule to exclude them from the transliteration:
<transform rule="[^åäöÅÄÖ] NFD"/><!-- do not undiactric some characters -->
Meh, sounds easy enough now!
Hopefully this will come in handy to someone.
Le 12/01/2017 à 02:51, David Cook a écrit :
Considering the folk at IndexData are Danish, perhaps they could provide some insight more directly.
David Cook Systems Librarian Prosentient Systems 72/330 Wattle St Ultimo, NSW 2007 Australia
Office: 02 9212 0899 Direct: 02 8005 0595
-----Original Message-----
From: koha-devel-bounces@lists.koha-community.org [mailto:koha-devel- bounces@lists.koha-community.org] On Behalf Of Colin Campbell Sent: Friday, 25 November 2016 3:32 AM To: koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] proper sorting order and search for Swedish characters and ISBN search
On Fri, Nov 04, 2016 at 05:20:03PM +0100, Gaetan Boisson wrote:
Hello,
I have been trying several things to try to sort this out, and haven't found a way (yet) that would avoid ugly compromises.
The issue is: in Swedish ä, ö and å are separate letters. Not variants of a and o. This means that searching for å shouldn't bring up a. When sorting, they belong to the very end of the alphabet, after z, not along
a and
o. In ICU, setting the locale parameter to "sv" (swedish) should do this. I
cant
seem to get this to work in yaz-icu but there are tests in the yaz code
base for
passing locale to strings. However I'm not sure how to correctly specify
this for
yaz the documentation is way too sketchy.
Colin
-- Colin Campbell Chief Software Engineer, PTFS Europe Limited Content Management and Library Solutions +44 (0) 800 756 6803 (phone) +44 (0) 7759 633626 (mobile) colin.campbell@ptfs-europe.com skype: colin_campbell2
http://www.ptfs-europe.com _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git :
http://git.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Gaetan Boisson Chef de projet bibliothécaire BibLibre +33(0)6 52 42 51 29 108 rue Breteuil 13006 Marseille gaetan.boisson@biblibre.com
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
participants (6)
-
Barton Chittenden -
Colin Campbell -
David Cook -
Gaetan Boisson -
Katrin -
Tajoli Zeno