- Koha-bugs - lists.koha-community.org

[Bug 15187] Adding 880 Fields to index-list in order to Increase Search for ALL non-latin Scripts
by bugzilla-daemon＠bugs.koha-community.org 02 May '22

02 May '22

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15187 --- Comment #19 from David Cook <dcook(a)prosentient.com.au> --- Created attachment 134457 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=134457&action=e… Bug 15187: Index 880 in Zebra the same as Elasticsearch This patch modifies the MARC21 export to Zebra, so that 880 fields are rewritten as their linked fields, in the same way that we already do with Elasticsearch, so that the alternate graphic representation of fields are indexed accordingly. (ie 880 $6245-01 Chinese titles will be indexed into the title index using the 245 rules) Test plan: 0. Apply patch 1. Turn on ICU indexing 1b. vi /etc/koha/zebradb/etc/default.idx 1c. Replace charmap word-phrase-utf.chr with icuchain words-icu.xml 1d. Replace charmap word-phrase-utf.chr with icuchain phrases-icu.xml 1e. Restart Zebra server 1f. Re-index Zebra 2. Add record with a 880 $6 245-01 $a 教牧書信 field. 3. Search for this record using a title index with the Chinese title 4. Note that the record is correctly retrieved (Note: This test probably works better using author or series as they present as links on the detail page which makes the fix more obviously useful.) -- You are receiving this mail because: You are watching all bug changes.

1 0

[Bug 15187] Adding 880 Fields to index-list in order to Increase Search for ALL non-latin Scripts
by bugzilla-daemon＠bugs.koha-community.org 02 May '22

02 May '22

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15187 David Cook <dcook(a)prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Discussion |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.

1 0

[Bug 15187] Adding 880 Fields to index-list in order to Increase Search for ALL non-latin Scripts
by bugzilla-daemon＠bugs.koha-community.org 02 May '22

02 May '22

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15187 David Cook <dcook(a)prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|gmcharlt(a)gmail.com |dcook(a)prosentient.com.au -- You are receiving this mail because: You are watching all bug changes.

1 0

[Bug 15187] Adding 880 Fields to index-list in order to Increase Search for ALL non-latin Scripts
by bugzilla-daemon＠bugs.koha-community.org 02 May '22

02 May '22

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15187 --- Comment #18 from David Cook <dcook(a)prosentient.com.au> --- (In reply to David Cook from comment #17) > Of course, this transform is MARC21 specific, so it needs to be done in a > way that is compatible with UNIMARC... I suppose another way to do this would be to change ./misc/migration_tools/rebuild_zebra.pl to use the same logic as ./Koha/SearchEngine/Elasticsearch.pm... -- You are receiving this mail because: You are watching all bug changes.

1 0

[Bug 15187] Adding 880 Fields to index-list in order to Increase Search for ALL non-latin Scripts
by bugzilla-daemon＠bugs.koha-community.org 02 May '22

02 May '22

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15187 --- Comment #17 from David Cook <dcook(a)prosentient.com.au> --- Of course, this transform is MARC21 specific, so it needs to be done in a way that is compatible with UNIMARC... -- You are receiving this mail because: You are watching all bug changes.

1 0

[Bug 15187] Adding 880 Fields to index-list in order to Increase Search for ALL non-latin Scripts
by bugzilla-daemon＠bugs.koha-community.org 02 May '22

02 May '22

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15187 --- Comment #16 from David Cook <dcook(a)prosentient.com.au> --- By the way, it looks like our indexing code for Elasticsearch already handles 880s: my $tag = $field->tag(); # Handle alternate scripts in MARC 21 my $altscript = 0; if ($marcflavour eq 'marc21' && $tag eq '880') { my $sub6 = $field->subfield('6'); if ($sub6 =~ /^(...)-\d+/) { $tag = $1; $altscript = 1; } } In that case, it looks like they're not indexing 880 fields at all. Perhaps that makes sense since if you index the 880 and the linkage derived from it, you'd end up with 2x the keyword indexed fields... -- You are receiving this mail because: You are watching all bug changes.

1 0

[Bug 15187] Adding 880 Fields to index-list in order to Increase Search for ALL non-latin Scripts
by bugzilla-daemon＠bugs.koha-community.org 02 May '22

02 May '22

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15187 David Cook <dcook(a)prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |In Discussion --- Comment #15 from David Cook <dcook(a)prosentient.com.au> --- The results are amazing! It works exactly like I hoped! I'm looking at a English/Chinese record that has many 880s for a variety of different fields. While Koha's default XSLTs will surface the 880 data on the detail page, *without this change*, the links don't work. However, with this change, they all work as you'd expect! Whether you click on the English or the Chinese, it's very smooth. -- Note that I haven't benchmarked this in a significant way. However, indexing 400 records in koha-testing-docker... there's no observable difference. -- You are receiving this mail because: You are watching all bug changes.

1 0

[Bug 15187] Adding 880 Fields to index-list in order to Increase Search for ALL non-latin Scripts
by bugzilla-daemon＠bugs.koha-community.org 02 May '22

02 May '22

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15187 --- Comment #14 from David Cook <dcook(a)prosentient.com.au> --- (In reply to David Cook from comment #13) > <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" > xmlns:marc="http://www.loc.gov/MARC21/slim" > version="1.0"> > <xsl:output indent="yes" > method="xml" > version="1.0" > encoding="UTF-8"/> > > <xsl:template name="identity" match="node()|@*"> > <xsl:copy> > <xsl:apply-templates select="@*|node()"/> > </xsl:copy> > </xsl:template> > > <xsl:template match="marc:datafield[@tag=880]"> > <xsl:call-template name="identity"/> > <xsl:variable name="tag" select="substring(marc:subfield[@code=6],1,3)"/> > <xsl:text>
</xsl:text> > <xsl:text> </xsl:text> > <xsl:element name="datafield" namespace="http://www.loc.gov/MARC21/slim"> > <xsl:attribute name="tag"><xsl:value-of > select="substring(marc:subfield[@code=6],1,3)"/></xsl:attribute> > <xsl:attribute name="ind1"><xsl:value-of > select="@ind1"/></xsl:attribute> > <xsl:attribute name="ind2"><xsl:value-of > select="@ind2"/></xsl:attribute> > <xsl:apply-templates select="marc:subfield[@code != '6']"/> > </xsl:element> > </xsl:template> > > </xsl:stylesheet> Steps to get this to work in a instance-specific context: 1. I saved the above as /etc/koha/zebradb/xsl/preprocess_marcxml.xsl 2. cp /etc/koha/zebradb/biblios/etc/dom-config.xml /etc/koha/sites/kohadev/test-dom-config.xml 3. vi /etc/koha/sites/kohadev/zebra-biblios-dom.cfg #For marcxml, I replaced /etc/koha/zebradb/biblios/etc/dom-config.xml with /etc/koha/sites/kohadev/test-dom-config.xml 4. vi /etc/koha/sites/kohadev/test-dom-config.xml #Use the following: <dom> <extract name="index"> <xslt stylesheet="preprocess_marcxml.xsl"/> <xslt stylesheet="biblio-zebra-indexdefs.xsl"/> </extract> <retrieve name="marc"> <xslt stylesheet="identity.xsl" /> </retrieve> <retrieve name="marcxml"> <xslt stylesheet="identity.xsl" /> </retrieve> <retrieve name="zebra"> <xslt stylesheet="zebra.xsl"/> </retrieve> <retrieve name="index"> <xslt stylesheet="biblio-zebra-indexdefs.xsl"/> </retrieve> <input> <xmlreader level="1"/> </input> </dom> 5. koha-rebuild-zebra -f -b -v kohadev -- You are receiving this mail because: You are watching all bug changes.

1 0

[Bug 15187] Adding 880 Fields to index-list in order to Increase Search for ALL non-latin Scripts
by bugzilla-daemon＠bugs.koha-community.org 02 May '22

02 May '22

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15187 --- Comment #13 from David Cook <dcook(a)prosentient.com.au> --- It's a bit of a hack and I haven't tried it yet with the Zebra pipelines yet, but I'll include my transform here. It copies all the existing XML elements, but for 880 elements it creates a new element based off the linkage, and it just copies the indicators and subfields into that new element according to the MARC bib spec. I would've liked to have integrated this into Koha's existing XSLTs but I couldn't see an easy way of doing so, although I have a few ideas about how maybe it could work... <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:marc="http://www.loc.gov/MARC21/slim" version="1.0"> <xsl:output indent="yes" method="xml" version="1.0" encoding="UTF-8"/> <xsl:template name="identity" match="node()|@*"> <xsl:copy> <xsl:apply-templates select="@*|node()"/> </xsl:copy> </xsl:template> <xsl:template match="marc:datafield[@tag=880]"> <xsl:call-template name="identity"/> <xsl:variable name="tag" select="substring(marc:subfield[@code=6],1,3)"/> <xsl:text>
</xsl:text> <xsl:text> </xsl:text> <xsl:element name="datafield" namespace="http://www.loc.gov/MARC21/slim"> <xsl:attribute name="tag"><xsl:value-of select="substring(marc:subfield[@code=6],1,3)"/></xsl:attribute> <xsl:attribute name="ind1"><xsl:value-of select="@ind1"/></xsl:attribute> <xsl:attribute name="ind2"><xsl:value-of select="@ind2"/></xsl:attribute> <xsl:apply-templates select="marc:subfield[@code != '6']"/> </xsl:element> </xsl:template> </xsl:stylesheet> -- You are receiving this mail because: You are watching all bug changes.

1 0

[Bug 15187] Adding 880 Fields to index-list in order to Increase Search for ALL non-latin Scripts
by bugzilla-daemon＠bugs.koha-community.org 02 May '22

02 May '22

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15187 --- Comment #12 from David Cook <dcook(a)prosentient.com.au> --- (In reply to David Cook from comment #11) > Although I wonder if I can think of something else clever... Another thought is to do more transformations in the Zebra DOM pipeline... either by adding an "input" pipeline or by adding more "Extract" transform documents. (See https://software.indexdata.com/zebra/doc/idzebra.pdf) The idea would be that the 880 $6 245-01 would be copied into a 245 and that way it could just be fed through the normal indexing process. Another thing to think about is how this is Zebra-specific. I don't know what would need to be done for Elasticsearch (if anything)... -- You are receiving this mail because: You are watching all bug changes.

1 0