May 2022 - Koha-bugs - lists.koha-community.org

[Bug 15187] Adding 880 Fields to index-list in order to Increase Search for ALL non-latin Scripts
by bugzilla-daemon＠bugs.koha-community.org 02 May '22

02 May '22

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15187 David Cook <dcook(a)prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|gmcharlt(a)gmail.com |dcook(a)prosentient.com.au -- You are receiving this mail because: You are watching all bug changes.

1 0

[Bug 15187] Adding 880 Fields to index-list in order to Increase Search for ALL non-latin Scripts
by bugzilla-daemon＠bugs.koha-community.org 02 May '22

02 May '22

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15187 --- Comment #18 from David Cook <dcook(a)prosentient.com.au> --- (In reply to David Cook from comment #17) > Of course, this transform is MARC21 specific, so it needs to be done in a > way that is compatible with UNIMARC... I suppose another way to do this would be to change ./misc/migration_tools/rebuild_zebra.pl to use the same logic as ./Koha/SearchEngine/Elasticsearch.pm... -- You are receiving this mail because: You are watching all bug changes.

1 0

[Bug 15187] Adding 880 Fields to index-list in order to Increase Search for ALL non-latin Scripts
by bugzilla-daemon＠bugs.koha-community.org 02 May '22

02 May '22

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15187 --- Comment #17 from David Cook <dcook(a)prosentient.com.au> --- Of course, this transform is MARC21 specific, so it needs to be done in a way that is compatible with UNIMARC... -- You are receiving this mail because: You are watching all bug changes.

1 0

[Bug 15187] Adding 880 Fields to index-list in order to Increase Search for ALL non-latin Scripts
by bugzilla-daemon＠bugs.koha-community.org 02 May '22

02 May '22

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15187 --- Comment #16 from David Cook <dcook(a)prosentient.com.au> --- By the way, it looks like our indexing code for Elasticsearch already handles 880s: my $tag = $field->tag(); # Handle alternate scripts in MARC 21 my $altscript = 0; if ($marcflavour eq 'marc21' && $tag eq '880') { my $sub6 = $field->subfield('6'); if ($sub6 =~ /^(...)-\d+/) { $tag = $1; $altscript = 1; } } In that case, it looks like they're not indexing 880 fields at all. Perhaps that makes sense since if you index the 880 and the linkage derived from it, you'd end up with 2x the keyword indexed fields... -- You are receiving this mail because: You are watching all bug changes.

1 0

[Bug 15187] Adding 880 Fields to index-list in order to Increase Search for ALL non-latin Scripts
by bugzilla-daemon＠bugs.koha-community.org 02 May '22

02 May '22

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15187 David Cook <dcook(a)prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |In Discussion --- Comment #15 from David Cook <dcook(a)prosentient.com.au> --- The results are amazing! It works exactly like I hoped! I'm looking at a English/Chinese record that has many 880s for a variety of different fields. While Koha's default XSLTs will surface the 880 data on the detail page, *without this change*, the links don't work. However, with this change, they all work as you'd expect! Whether you click on the English or the Chinese, it's very smooth. -- Note that I haven't benchmarked this in a significant way. However, indexing 400 records in koha-testing-docker... there's no observable difference. -- You are receiving this mail because: You are watching all bug changes.

1 0

[Bug 15187] Adding 880 Fields to index-list in order to Increase Search for ALL non-latin Scripts
by bugzilla-daemon＠bugs.koha-community.org 02 May '22

02 May '22

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15187 --- Comment #14 from David Cook <dcook(a)prosentient.com.au> --- (In reply to David Cook from comment #13) > <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" > xmlns:marc="http://www.loc.gov/MARC21/slim" > version="1.0"> > <xsl:output indent="yes" > method="xml" > version="1.0" > encoding="UTF-8"/> > > <xsl:template name="identity" match="node()|@*"> > <xsl:copy> > <xsl:apply-templates select="@*|node()"/> > </xsl:copy> > </xsl:template> > > <xsl:template match="marc:datafield[@tag=880]"> > <xsl:call-template name="identity"/> > <xsl:variable name="tag" select="substring(marc:subfield[@code=6],1,3)"/> > <xsl:text>
</xsl:text> > <xsl:text> </xsl:text> > <xsl:element name="datafield" namespace="http://www.loc.gov/MARC21/slim"> > <xsl:attribute name="tag"><xsl:value-of > select="substring(marc:subfield[@code=6],1,3)"/></xsl:attribute> > <xsl:attribute name="ind1"><xsl:value-of > select="@ind1"/></xsl:attribute> > <xsl:attribute name="ind2"><xsl:value-of > select="@ind2"/></xsl:attribute> > <xsl:apply-templates select="marc:subfield[@code != '6']"/> > </xsl:element> > </xsl:template> > > </xsl:stylesheet> Steps to get this to work in a instance-specific context: 1. I saved the above as /etc/koha/zebradb/xsl/preprocess_marcxml.xsl 2. cp /etc/koha/zebradb/biblios/etc/dom-config.xml /etc/koha/sites/kohadev/test-dom-config.xml 3. vi /etc/koha/sites/kohadev/zebra-biblios-dom.cfg #For marcxml, I replaced /etc/koha/zebradb/biblios/etc/dom-config.xml with /etc/koha/sites/kohadev/test-dom-config.xml 4. vi /etc/koha/sites/kohadev/test-dom-config.xml #Use the following: <dom> <extract name="index"> <xslt stylesheet="preprocess_marcxml.xsl"/> <xslt stylesheet="biblio-zebra-indexdefs.xsl"/> </extract> <retrieve name="marc"> <xslt stylesheet="identity.xsl" /> </retrieve> <retrieve name="marcxml"> <xslt stylesheet="identity.xsl" /> </retrieve> <retrieve name="zebra"> <xslt stylesheet="zebra.xsl"/> </retrieve> <retrieve name="index"> <xslt stylesheet="biblio-zebra-indexdefs.xsl"/> </retrieve> <input> <xmlreader level="1"/> </input> </dom> 5. koha-rebuild-zebra -f -b -v kohadev -- You are receiving this mail because: You are watching all bug changes.

1 0

[Bug 15187] Adding 880 Fields to index-list in order to Increase Search for ALL non-latin Scripts
by bugzilla-daemon＠bugs.koha-community.org 02 May '22

02 May '22

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15187 --- Comment #13 from David Cook <dcook(a)prosentient.com.au> --- It's a bit of a hack and I haven't tried it yet with the Zebra pipelines yet, but I'll include my transform here. It copies all the existing XML elements, but for 880 elements it creates a new element based off the linkage, and it just copies the indicators and subfields into that new element according to the MARC bib spec. I would've liked to have integrated this into Koha's existing XSLTs but I couldn't see an easy way of doing so, although I have a few ideas about how maybe it could work... <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:marc="http://www.loc.gov/MARC21/slim" version="1.0"> <xsl:output indent="yes" method="xml" version="1.0" encoding="UTF-8"/> <xsl:template name="identity" match="node()|@*"> <xsl:copy> <xsl:apply-templates select="@*|node()"/> </xsl:copy> </xsl:template> <xsl:template match="marc:datafield[@tag=880]"> <xsl:call-template name="identity"/> <xsl:variable name="tag" select="substring(marc:subfield[@code=6],1,3)"/> <xsl:text>
</xsl:text> <xsl:text> </xsl:text> <xsl:element name="datafield" namespace="http://www.loc.gov/MARC21/slim"> <xsl:attribute name="tag"><xsl:value-of select="substring(marc:subfield[@code=6],1,3)"/></xsl:attribute> <xsl:attribute name="ind1"><xsl:value-of select="@ind1"/></xsl:attribute> <xsl:attribute name="ind2"><xsl:value-of select="@ind2"/></xsl:attribute> <xsl:apply-templates select="marc:subfield[@code != '6']"/> </xsl:element> </xsl:template> </xsl:stylesheet> -- You are receiving this mail because: You are watching all bug changes.

1 0

[Bug 15187] Adding 880 Fields to index-list in order to Increase Search for ALL non-latin Scripts
by bugzilla-daemon＠bugs.koha-community.org 02 May '22

02 May '22

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15187 --- Comment #12 from David Cook <dcook(a)prosentient.com.au> --- (In reply to David Cook from comment #11) > Although I wonder if I can think of something else clever... Another thought is to do more transformations in the Zebra DOM pipeline... either by adding an "input" pipeline or by adding more "Extract" transform documents. (See https://software.indexdata.com/zebra/doc/idzebra.pdf) The idea would be that the 880 $6 245-01 would be copied into a 245 and that way it could just be fed through the normal indexing process. Another thing to think about is how this is Zebra-specific. I don't know what would need to be done for Elasticsearch (if anything)... -- You are receiving this mail because: You are watching all bug changes.

1 0

[Bug 15187] Adding 880 Fields to index-list in order to Increase Search for ALL non-latin Scripts
by bugzilla-daemon＠bugs.koha-community.org 02 May '22

02 May '22

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15187 --- Comment #11 from David Cook <dcook(a)prosentient.com.au> --- I suppose you could customize ./etc/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml so that the 880 fields mirror the potential linkages (like 245), but it would be very easy for them to fall out of sync. My method of using a more complex "match" attribute is easier to maintain, although it does still run into the problem of needing to add the 880 logic for each possible mapping, which is labour-intensive and error-prone. Jesse, your original idea has merit, but it won't work with how the Koha Zebra indexing XSLT has been designed. Although I wonder if I can think of something else clever... -- You are receiving this mail because: You are watching all bug changes.

1 0

[Bug 15187] Adding 880 Fields to index-list in order to Increase Search for ALL non-latin Scripts
by bugzilla-daemon＠bugs.koha-community.org 02 May '22

02 May '22

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15187 --- Comment #10 from David Cook <dcook(a)prosentient.com.au> --- Oh... actually this looks like it might be very difficult to do within the community model without making substantial changes to ./etc/zebradb/xsl/koha-indexdefs-to-zebra.xsl and ./etc/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml. These files were designed with too simple a logic in mind. I'm sorry but I don't have the resources to make those systematic architectural changes. I'm planning instead to locally customize ./etc/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl to have customized "match" attributes. For example: Replacing the following: <xslo:template mode="index_data_field" match="marc:datafield[@tag='245']"> With the following: <xslo:template mode="index_data_field" match="marc:datafield[@tag='245' or (@tag='880' and (marc:subfield[@code=6 and substring(text(),1,3)='245']))]"> While at a glance it's a very simple change, it's impossible to make that change using ./etc/zebradb/xsl/koha-indexdefs-to-zebra.xsl and ./etc/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml to create ./etc/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl -- You are receiving this mail because: You are watching all bug changes.

1 0