Linking bibs to authorities
Hi. I mentioned we also have a problem with linking bibs to authorities. We use authorities and want to link them to bibs. Over previous versions of Koha this has sometimes worked quite well. However on a few systems we have at version 3.12 it seems that the Zebra search into the authorities (used by link_bib_to_authorities) is returning all terms that contain any of the words in the bib field. So if the linkage is set to "default" then most authority searches will return multiple answers which means they do not link. If the linkage syspref is set to first or last then most of the bibs will link to the wrong authority as it is pot luck which will be first (or last) of the many authorities found by searching all the words ORed together. The only terms that will link are single-word authorities that are not used in any other authorities. If anyone has any pointers or knows how to get round or over this problem I would love to hear from you. More details on request... We are using Zebra dom indexing and icu chains and are using Koha 3.12 on Debian. We are not yet using QueryParser. Sysprefs and Help > About are here: Sysprefs are: +--------------------------+-------+ | variable | value | +--------------------------+-------+ | IncludeSeeFromInSearches | 0 | | QueryAutoTruncate | 0 | | QueryFuzzy | 0 | | QueryStemming | 0 | | QueryWeightFields | 1 | | TraceCompleteSubfields | 0 | | TraceSubjectSubdivisions | 0 | | UseICU | 1 | | UseQueryParser | 0 | +--------------------------+-------+ Koha> About: Koha version: 3.12.01.000 OS version ('uname -a'): Linux dclg.koha.ptfsadmin.uk0.bigv.io 3.2.0-4-amd64 #1 SMP Debian 3.2.46-1 x86_64 GNU/Linux Perl interpreter: /usr/bin/perl Perl version: 5.014002 Perl @INC: /home/koha/kohaclone /etc/perl /usr/local/lib/perl/5.14.2 /usr/local/share/perl/5.14.2 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.14 /usr/share/perl/5.14 /usr/local/lib/site_perl . MySQL version: mysql Ver 14.14 Distrib 5.5.31, for debian-linux-gnu (x86_64) using readline 6.2 Apache version: Server version: Apache/2.2.22 (Debian) Zebra version: Zebra 2.0.55 (C) 1994-2013, Index Data ApS Zebra is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. SHA1 ID: bd2bc9360225e695bbaba2c2d1cd6925c4eb23a5 Using ICU Many thanks folks. Ian -- Ian Bays Director of Projects, PTFS Europe Limited Content Management and Library Solutions +44 (0) 800 756 6803 (phone) +44 (0) 7774 995297 (mobile) +44 (0) 800 756 6384 (fax) skype: ian.bays email: ian.bays@ptfs-europe.com
Hie, It is because in Zebra, exact search does not work. It can find a phrase (all words in same order) but not say if this phrase fits the entire (sub)field. This is the purpose of PQF attribut 6, but is does not work. So when linking bibs to authorities, linker will match more authorities than needed. For example, in biblio record "History" will match authorities with headings "History", "History of art", "History of science", ... That is wy I created Bug 9072 : http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9072 Le 02/08/2013 17:32, Ian Bays a écrit :
Hi. I mentioned we also have a problem with linking bibs to authorities.
We use authorities and want to link them to bibs. Over previous versions of Koha this has sometimes worked quite well. However on a few systems we have at version 3.12 it seems that the Zebra search into the authorities (used by link_bib_to_authorities) is returning all terms that contain any of the words in the bib field.
So if the linkage is set to "default" then most authority searches will return multiple answers which means they do not link. If the linkage syspref is set to first or last then most of the bibs will link to the wrong authority as it is pot luck which will be first (or last) of the many authorities found by searching all the words ORed together. The only terms that will link are single-word authorities that are not used in any other authorities.
If anyone has any pointers or knows how to get round or over this problem I would love to hear from you. More details on request...
We are using Zebra dom indexing and icu chains and are using Koha 3.12 on Debian. We are not yet using QueryParser. Sysprefs and Help > About are here:
Sysprefs are:
+--------------------------+-------+ | variable | value | +--------------------------+-------+ | IncludeSeeFromInSearches | 0 | | QueryAutoTruncate | 0 | | QueryFuzzy | 0 | | QueryStemming | 0 | | QueryWeightFields | 1 | | TraceCompleteSubfields | 0 | | TraceSubjectSubdivisions | 0 | | UseICU | 1 | | UseQueryParser | 0 | +--------------------------+-------+
Koha> About:
Koha version: 3.12.01.000 OS version ('uname -a'): Linux dclg.koha.ptfsadmin.uk0.bigv.io 3.2.0-4-amd64 #1 SMP Debian 3.2.46-1 x86_64 GNU/Linux Perl interpreter: /usr/bin/perl Perl version: 5.014002 Perl @INC: /home/koha/kohaclone /etc/perl /usr/local/lib/perl/5.14.2 /usr/local/share/perl/5.14.2 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.14 /usr/share/perl/5.14 /usr/local/lib/site_perl . MySQL version: mysql Ver 14.14 Distrib 5.5.31, for debian-linux-gnu (x86_64) using readline 6.2 Apache version: Server version: Apache/2.2.22 (Debian) Zebra version: Zebra 2.0.55 (C) 1994-2013, Index Data ApS Zebra is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. SHA1 ID: bd2bc9360225e695bbaba2c2d1cd6925c4eb23a5 Using ICU
Many thanks folks. Ian
-- Fridolyn SOMERS Biblibre - Pôle support fridolyn.somers@biblibre.com
On Thu, Oct 17, 2013 at 05:53:57PM +0200, Fridolyn SOMERS wrote:
Hie,
It is because in Zebra, exact search does not work. It can find a phrase (all words in same order) but not say if this phrase fits the entire (sub)field.
Just an vague idea but I wonder if using zebra (or similar search utilities) for authorities is really a good idea. They are great for general bib search where you want to find more, even unexpected results. But with authorities your concern is to find a specific record as such the search facilities in a standard db are probably more useful. Just because a tool works well in one context it does not mean its the best in another (pick your favourite hammers & scredrivers analogy) Cheers Colin -- Colin Campbell Chief Software Engineer, PTFS Europe Limited Content Management and Library Solutions +44 (0) 800 756 6803 (phone) +44 (0) 7759 633626 (mobile) colin.campbell@ptfs-europe.com skype: colin_campbell2 http://www.ptfs-europe.com
Hi, On Thu, Oct 17, 2013 at 9:35 AM, Colin Campbell < colin.campbell@ptfs-europe.com> wrote:
Just an vague idea but I wonder if using zebra (or similar search utilities) for authorities is really a good idea. They are great for general bib search where you want to find more, even unexpected results. But with authorities your concern is to find a specific record as such the search facilities in a standard db are probably more useful. Just because a tool works well in one context it does not mean its the best in another (pick your favourite hammers & scredrivers analogy)
Well, the original motivation for implementing the Zebra DOM filter for MARC21 authority records was to enable the construction of normalized index strings for authority headings that included details such as the thesaurus for subject headings, thereby allowing precise matching that went beyond just the equivalent of keyword searches. Of course, the equivalent could be done in a database, but I think it would be fruitful to first properly investigate the bug. To that end, I request that a bug report actually be filed -- I see nothing apropos on Bugzilla -- preferably with sample records. Regards, Galen -- Galen Charlton Manager of Implementation Equinox Software, Inc. / The Open Source Experts email: gmc@esilibrary.com direct: +1 770-709-5581 cell: +1 404-984-4366 skype: gmcharlt web: http://www.esilibrary.com/ Supporting Koha and Evergreen: http://koha-community.org & http://evergreen-ils.org
Hi Galen, The patch in question that Ian mentions regarding adding phrases config to zebra for dom with icu.. which also heavily affects the authority linking results that he is talking about is under bug: 10729. I'm just about to update the bug with a description so it can be seen the two things relate.. The patch affects more than just the authorities linking script, but I believe using the linking script is possibly the easiest way to show that the patch is actually doing somthing. Just chipping in, Martin On 17 Oct 2013 10:17, "Galen Charlton" <gmc@esilibrary.com> wrote:
Hi,
On Thu, Oct 17, 2013 at 9:35 AM, Colin Campbell < colin.campbell@ptfs-europe.com> wrote:
Just an vague idea but I wonder if using zebra (or similar search utilities) for authorities is really a good idea. They are great for general bib search where you want to find more, even unexpected results. But with authorities your concern is to find a specific record as such the search facilities in a standard db are probably more useful. Just because a tool works well in one context it does not mean its the best in another (pick your favourite hammers & scredrivers analogy)
Well, the original motivation for implementing the Zebra DOM filter for MARC21 authority records was to enable the construction of normalized index strings for authority headings that included details such as the thesaurus for subject headings, thereby allowing precise matching that went beyond just the equivalent of keyword searches.
Of course, the equivalent could be done in a database, but I think it would be fruitful to first properly investigate the bug.
To that end, I request that a bug report actually be filed -- I see nothing apropos on Bugzilla -- preferably with sample records.
Regards,
Galen -- Galen Charlton Manager of Implementation Equinox Software, Inc. / The Open Source Experts email: gmc@esilibrary.com direct: +1 770-709-5581 cell: +1 404-984-4366 skype: gmcharlt web: http://www.esilibrary.com/ Supporting Koha and Evergreen: http://koha-community.org & http://evergreen-ils.org
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
Hi Your discussion is beyond my understanding of Zebra, but if some changes are needed to marcflavor specific conf files, please consider not doing the changes only for MARC21. Now we have a freshly working UNIMARC DOM indexing, bugs about DOM must also take UNIMARC into consideration if it is relevant. Regards Mathieu Le 17/10/2013 20:06, Martin Renvoize a écrit :
Hi Galen,
The patch in question that Ian mentions regarding adding phrases config to zebra for dom with icu.. which also heavily affects the authority linking results that he is talking about is under bug: 10729.
I'm just about to update the bug with a description so it can be seen the two things relate.. The patch affects more than just the authorities linking script, but I believe using the linking script is possibly the easiest way to show that the patch is actually doing somthing.
Just chipping in,
Martin
On 17 Oct 2013 10:17, "Galen Charlton" <gmc@esilibrary.com <mailto:gmc@esilibrary.com>> wrote:
Hi,
On Thu, Oct 17, 2013 at 9:35 AM, Colin Campbell <colin.campbell@ptfs-europe.com <mailto:colin.campbell@ptfs-europe.com>> wrote:
Just an vague idea but I wonder if using zebra (or similar search utilities) for authorities is really a good idea. They are great for general bib search where you want to find more, even unexpected results. But with authorities your concern is to find a specific record as such the search facilities in a standard db are probably more useful. Just because a tool works well in one context it does not mean its the best in another (pick your favourite hammers & scredrivers analogy)
Well, the original motivation for implementing the Zebra DOM filter for MARC21 authority records was to enable the construction of normalized index strings for authority headings that included details such as the thesaurus for subject headings, thereby allowing precise matching that went beyond just the equivalent of keyword searches.
Of course, the equivalent could be done in a database, but I think it would be fruitful to first properly investigate the bug.
To that end, I request that a bug report actually be filed -- I see nothing apropos on Bugzilla -- preferably with sample records.
Regards,
Galen -- Galen Charlton Manager of Implementation Equinox Software, Inc. / The Open Source Experts email: gmc@esilibrary.com <mailto:gmc@esilibrary.com> direct: +1 770-709-5581 <tel:%2B1%20770-709-5581> cell: +1 404-984-4366 <tel:%2B1%20404-984-4366> skype: gmcharlt web: http://www.esilibrary.com/ Supporting Koha and Evergreen: http://koha-community.org & http://evergreen-ils.org
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org <mailto:Koha-devel@lists.koha-community.org> http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Mathieu Saby Service d'Informatique Documentaire Service Commun de Documentation Université Rennes 2 Téléphone : 02 99 14 12 65 Courriel : mathieu.saby@univ-rennes2.fr
Hi, On Thu, Oct 17, 2013 at 1:10 PM, Mathieu Saby <mathieu.saby@univ-rennes2.fr>wrote:
Your discussion is beyond my understanding of Zebra, but if some changes are needed to marcflavor specific conf files, please consider not doing the changes only for MARC21. Now we have a freshly working UNIMARC DOM indexing, bugs about DOM must also take UNIMARC into consideration if it is relevant.
Agreed, and some of the techniques used to generate heading match indexes for MARC21 using the DOM filter can be applied to UNIMARC. Is there some documentation online about rules for matching bib headings to authority headings in a UNIMARC context? I assume that there would be points of similarity with MARC21, but I don't want to assume too much. Regards, Galen -- Galen Charlton Manager of Implementation Equinox Software, Inc. / The Open Source Experts email: gmc@esilibrary.com direct: +1 770-709-5581 cell: +1 404-984-4366 skype: gmcharlt web: http://www.esilibrary.com/ Supporting Koha and Evergreen: http://koha-community.org & http://evergreen-ils.org
Taking a quick look at Colin's patch, I don't see anything MARC21 specific in it.. It's mostly an addition to the MakeFile.pl to add the a default phrases-icu.xml config file to the system (whether that be MARC21, UNIMARK or even NORMARK). It's a simple patch which doesn't touch the linking scripts.. The linking script is just a great way to prove it's validity.. On 17 Oct 2013 13:29, "Galen Charlton" <gmc@esilibrary.com> wrote:
Hi,
On Thu, Oct 17, 2013 at 1:10 PM, Mathieu Saby < mathieu.saby@univ-rennes2.fr> wrote:
Your discussion is beyond my understanding of Zebra, but if some changes are needed to marcflavor specific conf files, please consider not doing the changes only for MARC21. Now we have a freshly working UNIMARC DOM indexing, bugs about DOM must also take UNIMARC into consideration if it is relevant.
Agreed, and some of the techniques used to generate heading match indexes for MARC21 using the DOM filter can be applied to UNIMARC.
Is there some documentation online about rules for matching bib headings to authority headings in a UNIMARC context? I assume that there would be points of similarity with MARC21, but I don't want to assume too much.
Regards,
Galen -- Galen Charlton Manager of Implementation Equinox Software, Inc. / The Open Source Experts email: gmc@esilibrary.com direct: +1 770-709-5581 cell: +1 404-984-4366 skype: gmcharlt web: http://www.esilibrary.com/ Supporting Koha and Evergreen: http://koha-community.org & http://evergreen-ils.org
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
Hi Fridolyn, Thank you for this. I was beginning to wonder if nobody was interested in the question. The bug you mention is a useful addition, but we achieved considerable success by correcting the contents of the default.idx (~/koha-dev/etc/zebradb/etc/default.idx) to use phrases-icu.xml for index p, and to ensure phrases-icu.xml and words-icu.xml have the correct stanzas for searching accent-blind. With these both in place I believe we have a very good hit rate for linking bib to authorities as long as the authority terms have been de-duplicated. I think Colin has submitted patches so that these are in place for all moving forwards, but it is good to know that others are using these features. What we were seeing before was if the term in the bib was (say) "History of Art" it would only match on the first word so would match an authority of (say) "History Books". All the best. Ian On 17/10/2013 16:53, Fridolyn SOMERS wrote:
Hie,
It is because in Zebra, exact search does not work. It can find a phrase (all words in same order) but not say if this phrase fits the entire (sub)field. This is the purpose of PQF attribut 6, but is does not work. So when linking bibs to authorities, linker will match more authorities than needed. For example, in biblio record "History" will match authorities with headings "History", "History of art", "History of science", ...
That is wy I created Bug 9072 : http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9072
Le 02/08/2013 17:32, Ian Bays a écrit :
Hi. I mentioned we also have a problem with linking bibs to authorities.
We use authorities and want to link them to bibs. Over previous versions of Koha this has sometimes worked quite well. However on a few systems we have at version 3.12 it seems that the Zebra search into the authorities (used by link_bib_to_authorities) is returning all terms that contain any of the words in the bib field.
So if the linkage is set to "default" then most authority searches will return multiple answers which means they do not link. If the linkage syspref is set to first or last then most of the bibs will link to the wrong authority as it is pot luck which will be first (or last) of the many authorities found by searching all the words ORed together. The only terms that will link are single-word authorities that are not used in any other authorities.
If anyone has any pointers or knows how to get round or over this problem I would love to hear from you. More details on request...
We are using Zebra dom indexing and icu chains and are using Koha 3.12 on Debian. We are not yet using QueryParser. Sysprefs and Help > About are here:
Sysprefs are:
+--------------------------+-------+ | variable | value | +--------------------------+-------+ | IncludeSeeFromInSearches | 0 | | QueryAutoTruncate | 0 | | QueryFuzzy | 0 | | QueryStemming | 0 | | QueryWeightFields | 1 | | TraceCompleteSubfields | 0 | | TraceSubjectSubdivisions | 0 | | UseICU | 1 | | UseQueryParser | 0 | +--------------------------+-------+
Koha> About:
Koha version: 3.12.01.000 OS version ('uname -a'): Linux dclg.koha.ptfsadmin.uk0.bigv.io 3.2.0-4-amd64 #1 SMP Debian 3.2.46-1 x86_64 GNU/Linux Perl interpreter: /usr/bin/perl Perl version: 5.014002 Perl @INC: /home/koha/kohaclone /etc/perl /usr/local/lib/perl/5.14.2 /usr/local/share/perl/5.14.2 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.14 /usr/share/perl/5.14 /usr/local/lib/site_perl . MySQL version: mysql Ver 14.14 Distrib 5.5.31, for debian-linux-gnu (x86_64) using readline 6.2 Apache version: Server version: Apache/2.2.22 (Debian) Zebra version: Zebra 2.0.55 (C) 1994-2013, Index Data ApS Zebra is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. SHA1 ID: bd2bc9360225e695bbaba2c2d1cd6925c4eb23a5 Using ICU
Many thanks folks. Ian
-- Ian Bays Director of Projects, PTFS Europe Limited Content Management and Library Solutions +44 (0) 800 756 6803 (phone) +44 (0) 7774 995297 (mobile) +44 (0) 800 756 6384 (fax) skype: ian.bays email: ian.bays@ptfs-europe.com
Hie, phrases-icu.xml ? this does not exist in current master. Could you give it ? Regards, Le 17/10/2013 19:06, Ian Bays a écrit :
Hi Fridolyn, Thank you for this. I was beginning to wonder if nobody was interested in the question.
The bug you mention is a useful addition, but we achieved considerable success by correcting the contents of the default.idx (~/koha-dev/etc/zebradb/etc/default.idx) to use phrases-icu.xml for index p, and to ensure phrases-icu.xml and words-icu.xml have the correct stanzas for searching accent-blind.
With these both in place I believe we have a very good hit rate for linking bib to authorities as long as the authority terms have been de-duplicated.
I think Colin has submitted patches so that these are in place for all moving forwards, but it is good to know that others are using these features.
What we were seeing before was if the term in the bib was (say) "History of Art" it would only match on the first word so would match an authority of (say) "History Books".
All the best.
Ian On 17/10/2013 16:53, Fridolyn SOMERS wrote:
Hie,
It is because in Zebra, exact search does not work. It can find a phrase (all words in same order) but not say if this phrase fits the entire (sub)field. This is the purpose of PQF attribut 6, but is does not work. So when linking bibs to authorities, linker will match more authorities than needed. For example, in biblio record "History" will match authorities with headings "History", "History of art", "History of science", ...
That is wy I created Bug 9072 : http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9072
Le 02/08/2013 17:32, Ian Bays a écrit :
Hi. I mentioned we also have a problem with linking bibs to authorities.
We use authorities and want to link them to bibs. Over previous versions of Koha this has sometimes worked quite well. However on a few systems we have at version 3.12 it seems that the Zebra search into the authorities (used by link_bib_to_authorities) is returning all terms that contain any of the words in the bib field.
So if the linkage is set to "default" then most authority searches will return multiple answers which means they do not link. If the linkage syspref is set to first or last then most of the bibs will link to the wrong authority as it is pot luck which will be first (or last) of the many authorities found by searching all the words ORed together. The only terms that will link are single-word authorities that are not used in any other authorities.
If anyone has any pointers or knows how to get round or over this problem I would love to hear from you. More details on request...
We are using Zebra dom indexing and icu chains and are using Koha 3.12 on Debian. We are not yet using QueryParser. Sysprefs and Help > About are here:
Sysprefs are:
+--------------------------+-------+ | variable | value | +--------------------------+-------+ | IncludeSeeFromInSearches | 0 | | QueryAutoTruncate | 0 | | QueryFuzzy | 0 | | QueryStemming | 0 | | QueryWeightFields | 1 | | TraceCompleteSubfields | 0 | | TraceSubjectSubdivisions | 0 | | UseICU | 1 | | UseQueryParser | 0 | +--------------------------+-------+
Koha> About:
Koha version: 3.12.01.000 OS version ('uname -a'): Linux dclg.koha.ptfsadmin.uk0.bigv.io 3.2.0-4-amd64 #1 SMP Debian 3.2.46-1 x86_64 GNU/Linux Perl interpreter: /usr/bin/perl Perl version: 5.014002 Perl @INC: /home/koha/kohaclone /etc/perl /usr/local/lib/perl/5.14.2 /usr/local/share/perl/5.14.2 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.14 /usr/share/perl/5.14 /usr/local/lib/site_perl . MySQL version: mysql Ver 14.14 Distrib 5.5.31, for debian-linux-gnu (x86_64) using readline 6.2 Apache version: Server version: Apache/2.2.22 (Debian) Zebra version: Zebra 2.0.55 (C) 1994-2013, Index Data ApS Zebra is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. SHA1 ID: bd2bc9360225e695bbaba2c2d1cd6925c4eb23a5 Using ICU
Many thanks folks. Ian
-- Fridolyn SOMERS Biblibre - Pôle support fridolyn.somers@biblibre.com
On Thu, Oct 24, 2013 at 04:18:54PM +0200, Fridolyn SOMERS wrote:
Hie,
phrases-icu.xml ? this does not exist in current master. Could you give it ?
Its in the patch for bug 10729 based on the one distributed with zebra. The patch also makes the default reference it Cheers Colin -- Colin Campbell Chief Software Engineer, PTFS Europe Limited Content Management and Library Solutions +44 (0) 800 756 6803 (phone) +44 (0) 7759 633626 (mobile) colin.campbell@ptfs-europe.com skype: colin_campbell2 http://www.ptfs-europe.com
participants (6)
-
Colin Campbell -
Fridolyn SOMERS -
Galen Charlton -
Ian Bays -
Martin Renvoize -
Mathieu Saby