[Bug 8984] New: Zebra does not index some fields in UNIMARC biblio records
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8984 Priority: P5 - low Change sponsored?: --- Bug ID: 8984 Assignee: gmcharlt@gmail.com Summary: Zebra does not index some fields in UNIMARC biblio records Severity: minor Classification: Unclassified OS: All Reporter: mathieu.saby@univ-rennes2.fr Hardware: All Status: NEW Version: rel_3_10 Component: Searching Product: Koha In marc_defs / unimarc / biblios /, some fields are missing in record.abs, biblio-koha-indexdefs.xml and biblio-zebra-indexdefs.xsl : - Identifier-standard : 011f/g ; 012a ; 013a/z ; 014a/z ; 015a/z ; 016a/z ; 017a/z, 040a/z, 071z, 072z, 073z - Authors : 716, 72X, 730 - Subjects : 608, 615, 616, 617, 620, 621 (and maybe 66X ?) - Title : 411t, 421-425t, 433-437t, 442-444t, 446-456t, 462-463t, 470-488t - Publisher : 071b (may contain the name of a music publisher) - Classification codes : 670, 675, 686 Beside : - 207 and 208 are wrong. - 4XX$d are useless : pubdate should be the date of the document itself, not of related titles. - some fields in record.abs do not exist in Unimarc standard : 414-420 ; 603, 630-636 First of all, I will make a patch for record.abs. If it is OK, I will make other patches for the 2 DOM configuration files. (And after that, I will make other patches to improve unimarc indexation) Regards, Mathieu Saby Rennes 2 University -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8984 mathieu saby <mathieu.saby@univ-rennes2.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8984 mathieu saby <mathieu.saby@univ-rennes2.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|gmcharlt@gmail.com |mathieu.saby@univ-rennes2.f | |r -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8984 --- Comment #1 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- Created attachment 13115 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=13115&action=edit [PATCH] Bug 8984: index new fields in UNIMARC record.abs This patch - add new (sub)fields to Identifier-standard index : 011f/g ; 012a ; 013a/z ; 014a/z ; 015a/z ; 016a/z ; 017a/z, 040a/z, 071z, 072z, 073z - add 1 new subfield to Publisher index : 071b (may contain the name of a music publisher) - add new (sub)fields to Author and Identifier-standard index (for the $9) : 716, 72X, 730 - correct 207 and 208 - suppress 308a and 328a in Note (useless as complete fields are indexed in same index) - add subfields to Title index : 411t, 421-425t, 433-437t, 442-444t, 446-456t, 462-463t, 470-488t - supress subfields from Pubdate index : 4XX$d (pubdate should be the date of the document itself, not of related titles. Besides, 4XX$d is not used in France) - add (sub)fields to Subject and Identifier-standard index (for the $9) : 608, 615, 616, 617, 620, 621 - add some classifications index : 670, 675, 686 - suppress some non unimarc fields (414-420 ; 603, 630-636) - add some comments (to make easier further modifications) To test : - take a record and fill some of the missing fields (e.g 488t, 608, 720, 012a) with some data as "field488", "field608" etc - try to find the record => not possible - apply the patch, copy the new record.abs in etc/zebradb/biblios/etc and rebuild zebra - try to find the record => should be ok - check nothing else is broken... If it is correct, I'll make an other patch on the same bug for the 2 DOM indexing configuration files. Mathieu Saby -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8984 mathieu saby <mathieu.saby@univ-rennes2.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8984 --- Comment #2 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- If some libraries are currently using non-unimarc fields (414-420 ; 603, 630-636) for some reason, I could edit the patch to keep these fields in record.abs, and just add a comment before each field to let people know it is not plain UNIMARC... Mathieu Saby -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8984 Jared Camins-Esakov <jcamins@cpbibliography.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |ASSIGNED --- Comment #3 from Jared Camins-Esakov <jcamins@cpbibliography.com> --- (In reply to comment #2)
If some libraries are currently using non-unimarc fields (414-420 ; 603, 630-636) for some reason, I could edit the patch to keep these fields in record.abs, and just add a comment before each field to let people know it is not plain UNIMARC...
I would prefer this option, especially so close to a release. I look forward to seeing your revised patch. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8984 Jared Camins-Esakov <jcamins@cpbibliography.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version|rel_3_10 |master -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8984 --- Comment #4 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- Created attachment 13269 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=13269&action=edit [PATCH] Bug 8984: index new fields in UNIMARC record.abs (revision 1) I reverted the suppression of non-unimarc fields. In this new patch they are just commented. (and I add 560 field that I did not see last week) M. Saby Rennes 2 University -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8984 mathieu saby <mathieu.saby@univ-rennes2.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #13115|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8984 --- Comment #5 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- Created attachment 13270 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=13270&action=edit [PATCH] Bug 8984: index new fields in UNIMARC DOM-indexing configuration files With the new record.abs, I created a new biblio-koha-indexdefs.xml with make_zebra_dom_cfg_from_record_abs, then a new biblio-zebra-indexdefs.xsl with xsltproc, and I made a patch. It must be applied with patch 13269 and tested in with Koha using DOM-indexing. M. Saby Rennes 2 University -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8984 mathieu saby <mathieu.saby@univ-rennes2.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8984 Jared Camins-Esakov <jcamins@cpbibliography.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|minor |normal -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8984 mathieu saby <mathieu.saby@univ-rennes2.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #13269|0 |1 is obsolete| | Attachment #13270|0 |1 is obsolete| | --- Comment #6 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- Created attachment 13303 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=13303&action=edit [PATCH] Bug 8984: make Zebra more UNIMARC compliant This new patch does not suppress the 4XX$d. This patch makes the following changes to record.abs, biblio-koha-indexdefs.xml and biblio-zebra-indexdefs.xsl : - adding new (sub)fields to Identifier-standard index : 011f/g ; 012a ; 013a/z ; 014a/z ; 015a/z ; 016a/z ; 017a/z, 040a/z, 071z, 072z, 073z - adding 1 new subfield to Publisher index : 071b (may contain the name of a music publisher) - adding new (sub)fields to Author and Identifier-standard index (for the $9) : 716, 72X, 730 - adding new (sub)fields to Note : 334$a (award note) - correcting 207 and 208 - suppressing 308a and 328a in Note (useless as complete fields are indexed in same index) - adding (sub)fields to Title index : 411t, 421-425t, 433-437t, 442-444t, 446-456t, 462-463t, 470-488t, 560 - adding (sub)fields to Subject and Identifier-standard index (for the $9) : 608, 615, 616, 617, 620, 621 - adding some classifications index : 670, 675, 686 - adding some comments (to make easier further modifications and to identify non unimarc fields : 414-420, 603, 630-636) To test : - take a record and fill some of the missing fields (e.g 488t, 608, 720, 012a) with some data as "field488", "field608" etc - try to find the record => not possible - apply the patch, copy the new record.abs in etc/zebradb/biblios/etc and rebuild zebra - try to find the record => should be ok - check nothing else is broken... Mathieu Saby Rennes 2 University -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8984 Zeno Tajoli <z.tajoli@cineca.it> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |z.tajoli@cineca.it Patch complexity|--- |Medium patch --- Comment #7 from Zeno Tajoli <z.tajoli@cineca.it> --- I see the path. For Italian users of Unimarc, it is OK. The patch doesn't change anything used by italian unimarc users. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8984 --- Comment #8 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- Thank you Zeno. So could you sign it off ? Mathieu Saby -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8984 Zeno Tajoli <z.tajoli@cineca.it> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Patch doesn't apply --- Comment #9 from Zeno Tajoli <z.tajoli@cineca.it> --- Hi Mathieu, the patch has problem on master.
From sandboxes:
Something went wrong ! Applying: Bug 8984: make Zebra more UNIMARC compliant Using index info to reconstruct a base tree... Falling back to patching base and 3-way merge... Auto-merging etc/zebradb/marc_defs/unimarc/biblios/biblio-koha-indexdefs.xml CONFLICT (content): Merge conflict in etc/zebradb/marc_defs/unimarc/biblios/biblio-koha-indexdefs.xml Auto-merging etc/zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl Failed to merge in the changes. Patch failed at 0001 Bug 8984: make Zebra more UNIMARC compliant When you have resolved this problem run git am --resolved. If you would prefer to skip this patch, instead run git am --skip. To restore the original branch and stop patching run git am --abort. Bug 8984 - Zebra does not index some fields in UNIMARC biblio records I think you need to rebase it -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8984 Paul Poulain <paul.poulain@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |paul.poulain@biblibre.com --- Comment #10 from Paul Poulain <paul.poulain@biblibre.com> --- (In reply to comment #9)
Hi Mathieu,
the patch has problem on master. From sandboxes:
Something went wrong ! Hi Zeno & Mathieu,
changes related to zebra configuration are *not* managed properly by the sandboxes. It must be done manually. Sorry about that. [ patches about updatedatabase can't too, but this problem will be fixed with the new updatedb mechanism (once it gets in...) ] -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8984 mathieu saby <mathieu.saby@univ-rennes2.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #13303|0 |1 is obsolete| | --- Comment #11 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- Created attachment 13899 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=13899&action=edit [PATCH] Bug 8984: make Zebra more UNIMARC compliant Patch rebased This patch makes the following changes to record.abs, biblio-koha-indexdefs.xml and biblio-zebra-indexdefs.xsl : - adding new (sub)fields to Identifier-standard index : 011f/g ; 012a ; 013a/z ; 014a/z ; 015a/z ; 016a/z ; 017a/z, 040a/z, 071z, 072z, 073z - adding 1 new subfield to Publisher index : 071b (may contain the name of a music publisher) - adding new (sub)fields to Author and Identifier-standard index (for the $9) : 716, 72X, 730 - adding new (sub)fields to Note : 334$a (award note) - correcting 207 and 208 - suppressing 308a and 328a in Note (useless as complete fields are indexed in same index) - adding (sub)fields to Title index : 411t, 421-425t, 433-437t, 442-444t, 446-456t, 462-463t, 470-488t, 560 - adding (sub)fields to Subject and Identifier-standard index (for the $9) : 608, 615, 616, 617, 620, 621 - adding some classifications index : 670, 675, 686 - adding some comments (to make easier further modifications and to identify non unimarc fields : 414-420, 603, 630-636, 646) To test : - take a record and fill some of the missing fields (e.g 488t, 608, 720, 012a) with some data as "field488", "field608" etc - try to find the record => not possible - apply the patch, copy the new record.abs in etc/zebradb/biblios/etc and rebuild zebra - try to find the record => should be ok - check nothing else is broken... - same test with DOM indexing activated -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8984 Zeno Tajoli <z.tajoli@cineca.it> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8984 --- Comment #12 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- (In fact I am probably wrong saying you need to copy the file in etc/zebradb/biblios/etc In my install, the file was in this directory in addition to etc/zebradb/marc_defs/unimarc/biblios) M. Saby -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8984 Zeno Tajoli <z.tajoli@cineca.it> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #13899|0 |1 is obsolete| | --- Comment #13 from Zeno Tajoli <z.tajoli@cineca.it> --- Created attachment 13965 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=13965&action=edit the entire patch sign-offed It apply correctly to current master -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8984 Zeno Tajoli <z.tajoli@cineca.it> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8984 Paul Poulain <paul.poulain@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA QA Contact| |paul.poulain@biblibre.com --- Comment #14 from Paul Poulain <paul.poulain@biblibre.com> --- QA comments: I made the following tests : * without patch = rm -rf $KOHA_CONF_DIR && perl Makefile.PL && make install with grs1 unimarc ==> indexing OK (can index & search) * with patch = rm -rf $KOHA_CONF_DIR && perl Makefile.PL && make install with grs1 unimarc ==> indexing OK (can index & search) * without patch = rm -rf $KOHA_CONF_DIR && perl Makefile.PL && make install with dom unimarc ==> indexing is KO, I get the following error message: 15:28:48-03/01 zebraidx(25369) [warn] Unknown register type: * after patch = rm -rf $KOHA_CONF_DIR && perl Makefile.PL && make install with grs1 unimarc ==> indexing is KO, I get the following error message: 15:12:31-03/01 zebraidx(22028) [warn] Unknown register type: It seems that the DOM indexing in UNIMARC has a problem, not related to this patch. I'll open a bug for this problem, and mark passed QA this patch. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8984 Paul Poulain <paul.poulain@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #13965|0 |1 is obsolete| | --- Comment #15 from Paul Poulain <paul.poulain@biblibre.com> --- Created attachment 14395 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=14395&action=edit Bug 8984: make Zebra more UNIMARC compliant This patch makes the following changes to record.abs, biblio-koha-indexdefs.xml and biblio-zebra-indexdefs.xsl : - adding new (sub)fields to Identifier-standard index : 011f/g ; 012a ; 013a/z ; 014a/z ; 015a/z ; 016a/z ; 017a/z, 040a/z, 071z, 072z, 073z - adding 1 new subfield to Publisher index : 071b (may contain the name of a music publisher) - adding new (sub)fields to Author and Identifier-standard index (for the $9) : 716, 72X, 730 - adding new (sub)fields to Note : 334$a (award note) - correcting 207 and 208 - suppressing 308a and 328a in Note (useless as complete fields are indexed in same index) - adding (sub)fields to Title index : 411t, 421-425t, 433-437t, 442-444t, 446-456t, 462-463t, 470-488t, 560 - adding (sub)fields to Subject and Identifier-standard index (for the $9) : 608, 615, 616, 617, 620, 621 - adding some classifications index : 670, 675, 686 - adding some comments (to make easier further modifications and to identify non unimarc fields : 414-420, 603, 630-636, 646) To test : - take a record and fill some of the missing fields (e.g 488t, 608, 720, 012a) with some data as "field488", "field608" etc - try to find the record => not possible - apply the patch, copy the new record.abs in etc/zebradb/biblios/etc and rebuild zebra - try to find the record => should be ok - check nothing else is broken... - same test with DOM indexing activated http://bugs.koha-community.org/show_bug.cgi?id=8984 Signed-off-by: Zeno Tajoli <tajoli@cilea.it> Signed-off-by: Paul Poulain <paul.poulain@biblibre.com> -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8984 --- Comment #16 from Paul Poulain <paul.poulain@biblibre.com> --- (In reply to comment #14)
It seems that the DOM indexing in UNIMARC has a problem, not related to this patch. I'll open a bug for this problem, and mark passed QA this patch.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9342 -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8984 Jared Camins-Esakov <jcamins@cpbibliography.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to Master --- Comment #17 from Jared Camins-Esakov <jcamins@cpbibliography.com> --- This patch has been pushed to master. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8984 Chris Cormack <chris@bigballofwax.co.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to Master |Pushed to Stable CC| |chris@bigballofwax.co.nz --- Comment #18 from Chris Cormack <chris@bigballofwax.co.nz> --- Pushed to 3.10.x and 3.8.x -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon@bugs.koha-community.org