http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8252 mathieu saby <mathieu.saby@univ-rennes2.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mathieu.saby@univ-rennes2.f | |r --- Comment #1 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- Oh yes, coded fields are very bad indexed in unimarc. One pbm is some index in unimarc don't match exaclty indexs in marc21. I have prepared a big patch for that (defining new indexs in CCL etc). I will send it to you by email, maybe it could help you. M. Saby -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8252 --- Comment #2 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- Created attachment 15007 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=15007&action=edit [PATCH 1/2] Bug 8252: Correct UNIMARC 1xx fields in DOM biblio and GRS-1 In DOM indexing, positions of biblio are not taken into account. Additionnaly some indexes are not working even in GRS1, and there are a lot of warns when rebuiding zebra. This patch correct all 1xx indexes in GRS1 and DOM, and suppress the warns. It also creates new indexes : Country-heading for 101, Code-langage-original for 102$c, Material-type for 200$b. 106$a (used for printed docs only) is no more indexed in itype but in ff8-23 instead. For the moment I keep 200$b indexed in itype and itemtype but if no library use this field that way. To test : Apply on a Koha using GRS1. Check if all coded fields are searchable in advanced search. Check in simple search if Material-type, Country heading and Code-langage-original are searchable (ex :"Material-type:texte" ; "Country-heading:FR" ; "Code-langage-original:eng") Apply on a Koha using DOM. Check if all coded fields are searchable in advanced search. Check in simple search if Material-type, Country heading and Code-langage-original are searchable (ex :"Material-type:texte" ; "Country-heading:FR" ; "Code-langage-original:eng") M. Saby Rennes 2 University -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8252 --- Comment #3 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- Created attachment 15008 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=15008&action=edit [PATCH 2/2] Bug 8252: Followup for Marc21 Followup for Marc21 To test : Apply on a Koha using GRS1. Check if all coded fields are searchable in advanced search. Check in simple search if Country heading and Code-langage-original are searchable (ex "Country-heading:FR" ; "Code-langage-original:eng") Apply on a Koha using DOM. same tests -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8252 mathieu saby <mathieu.saby@univ-rennes2.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|--- |Small patch -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8252 mathieu saby <mathieu.saby@univ-rennes2.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #15008|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8252 mathieu saby <mathieu.saby@univ-rennes2.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #15007|0 |1 is obsolete| | --- Comment #7 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- Created attachment 16945 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=16945&action=edit [PATCH 1/2] Bug 8252: Fix indexing of UNIMARC 1xx for GRS-1 Before fixing UNIMARC DOM indexing, we must fix GRS-1 indexing 1) In advanced search, some Coded fields index are not working : Print, Illustration, Content 2) Country-heading index is not working 3) Some subfields are indexed in wrong indexes : 102$a should be in Country-publication instead of Country-heading (non defined in bib1.att) 106$a, filled only for printed works, should be in ff88-23 (form of item) instead of itype (ff88-23 is made for Marc21 008 pos 23, which contains the same data as 106a) 200$b should be in Material-type instead of (or in addition to) itype and itemtype (Material-type :"free-form string, ... that describes the material type of the item, e.g., cassette, kit, computer database, computer file.") 100$a pos 22-24 should not be indexed as "ln" : it is the language of the record, not the language of the ressource 4) Index names are too long : if we index new positions of coded fields, with existing names it breaks zebra indexing (there must be a limit in line lenghth in record.abs?) 5) There are a lot of warns when rebuiding zebra. This patch make some changes in bib1.att (could be used later to improve search) : - fixing wording for att 51 and 1012 - adding comments for attributes based on MARC21 008 field (8800-8841) - creating 8806 (tpubdate), 8838 (Modified-code), 8818 (ff8-18), 8840 (ff8-18-21), 8819 (ff8-19), 8821 (ff8-21), 8828 (ff8-28), 8830 (ff8-30), 8831 (ff8-31) - creating attributes specific to UNIMARC : 9701-9707 (Video-mt, Graphics-type, Graphics-support, Title-page-availability, Cumulative-index-availability, script-Title, char-encoding) - setting apart 3 blocks of attributes, so it could be easy to make further changes : -- common to Marc21 and UNIMARC : 8806, 8822, 8838 -- slightly different in Marc21 and UNIMARC (different meanings according to the type of the record => don't match a single UNIMARC field) -- specific to UNIMARC : 9701-9707 in ccl.properties : - creating a new index: Country-publication 1=1053 - suppressing some warns by mapping with bib1 att: Date-time-last-modified, Name, rtype, Music-number - defining indexes using the 3 blocks attributes defined in bib1 (common to Marc21 and UNIMARC, slightly different, specific to UNIMARC) in record.abs : - renaming some index for 100-105-110 fields - correcting indexing of 102$a (country of publication) 106$a (ff88-23) 100$a pos 22-24 (language of record, no more indexed) 105$a pos. 0-3 (illustration code) 200$b (for the moment, I keep it indexed in itype and itemtype, but also Material-Type) in C4/Search.pm : - adding "Country-publication" index in opac and staff interface template subtypes_unimarc.in : - renaming indexes to take into account the changes made to Zebra config files To test (this cannot be done with a sandbox) : 1) apply the patch in a UNIMARC Koha 2) reindex your data (rebuild_zebra -x -b -r -v) 3) try to use those Coded fields indexes in Advanced search, in OPAC and Staff interface (available after clicking on "More options", then on "Coded information filters"): Audience, Print, Literary genre, Biography, Illustration, Content, Video Types, Serials, Serial Type, Periodicity, Regularity 4) try to search "Country-publication=FR" in simple search M. Saby -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8252 --- Comment #9 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- In 1st patch, changes made to templates files are maybe too important (no absolute need to changes some indexes names), but I can rewrite it. M. Saby -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8252 --- Comment #11 from Paul Poulain <paul.poulain@biblibre.com> --- hi Mathieu, Trying to investigate this patch, because it's a useful improvement I haven't tested it yet, but i'm QAing first. (disclaimer : when I say "MARC21", I mean MARC21 and NORMAC, that are very close) I find some strange things: * Country of publication + 'Country-publication', => you add this entry, just for UNIMARC. Isn't it relevant for MARC21 ? The LoC use Country-publication for 1053, so your choice is OK (http://www.loc.gov/z3950/agency/defns/bib1.html) * in MARC21, there's a use of Date/time-last-modified, not Date-time-last-modified : etc/zebradb/marc_defs/normarc/biblios/record.abs:# NiN melm 005 Date/time-last-modified etc/zebradb/marc_defs/marc21/biblios/record.abs:melm 005 Date/time-last-modified etc/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml: <!--record.abs line 44: melm 005 Date/time-last-modified--> etc/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml: <target_index>Date/time-last-modified:w</target_index> etc/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl: <z:index name="Date/time-last-modified:w"> => WHY this change ? * music publisher: you change -att 51 Number-music-publisher +att 51 Music-number but number-music-publisher is used by MARC21, so why do you remove it ? I think we could/should use Identifier-publisher-for-music everywhere, as it's bib1 attribute name for LoC (http://www.loc.gov/z3950/agency/bib1.html) I have no other comments, except a general one = it would have been much easier to test this patch if it had been splitted in smaller ones. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8252 mathieu saby <mathieu.saby@univ-rennes2.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #16945|0 |1 is obsolete| | --- Comment #13 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- Created attachment 17970 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=17970&action=edit [PATCH 1] Bug 8252: Fix indexing of UNIMARC 1xx for GRS-1 -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8252 --- Comment #15 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- Created attachment 17972 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=17972&action=edit [PATCH 3] Bug 8252 : Followup for Date/time-last-modified and Music number This followup restores the original wording of "Date/time-last-modified" index, and change the name of "Music-number" index to "Number-music-publisher" To test : 1. In a UNIMARC Koha instance 2. apply patchs #1, #2 and this followup 3. copy from src/etc/zebradb directory to the etc/zebradb/ in your main koha directory the following files : -- zebradb/biblios/etc/bib1.att -- zebradb/ccl.properties -- zebradb/marc_defs/unimarc/biblios/record.abs -- zebradb/marc_defs/unimarc/biblios/biblio-koha-indexdefs.xml -- zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl 4. rebuild zebra with -b -x -v -r options 5. write a value like "test071a" in 071$a field in a record 6. check if you can find this record with this search : "ccl=Number-music-publisher:test071a" -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8252 --- Comment #17 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- There is a warn when you rebuild zebra, but I think it is not linked with my patches (22:03:02-06/05 zebraidx(13748) [warn] Unknown register type: ) M. Saby -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8252 Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #17971|0 |1 is obsolete| | --- Comment #19 from Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> --- Created attachment 21058 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=21058&action=edit [SIGNED-OFF] Bug 8252: Fix indexing of UNIMARC 1xx for DOM This patch makes the same changes in UNIMARC DOM configuration as PATCH 1 made for GRS-1. You MUST apply the first patch with this one (this one depends on some modifications made by patch 1/2) positions of subfields are indexed that way : In biblio-koha-indexdefs.xml : tag="100" subfields="a" offset="17" length="1" In biblio-zebra-indexdefs.xsl : xslo:value-of select="substring(., 17, 1)" I had to edit biblio-zebra-indexdefs.xsl by hand, because etc/zebdradb/xml/koha-indexdefs-to-zebra.xsl does only support "subtring" in handle-one-index-control-field template. It is good for MARC21, but not for UNIMARC : in MARC21, indexing subtrings is needed for controled field (001-009, with no subfields) But in UNIMARC it is needed for subfields of 1XX fields. So if DOM indexing is working with these new files, we may need to change koha-indexdefs-to-zebra.xsl. Test plan (not possible in a sandbox) : 1) In a Koha instance using UNIMARC and DOM indexing 2) apply Patch 1 and Patch 2 (this one) 3) copy the following files from the etc/zebradb directory of your source into the etc/zebradb directory of your main Koha directory : -- etc/zebradb/marc_defs/unimarc/biblios/biblio-koha-indexdefs.xml -- etc/zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl -- etc/zebradb/ccl.properties -- etc/zebradb/biblios/etc/bib1.att 4) rebuild zebra with -x -b -r -v options 5) check if coded filters in advanced search are usable in OPAC and Staff interface Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Works. No koha-qa errors. Test for DOM Apply patches Don't forget to copy files reindex Search by coded fields works, also Country-publication -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8252 Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off CC| |bgkriegel@gmail.com --- Comment #21 from Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> --- Long to test, but works. But I would be very happy if we have a sample of UNIMARC bib record with its authorities/auth types for testing ready to be imported on standar dev install. A small but known dataset could made search testing easier. Good work! -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8252 --- Comment #23 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- By the way, if tests and QA is OK, further work will be needed to change the behavior of koha-indexdefs-to-zebra.xsl (as explained in the commit message of attachment 21058. M. Saby -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8252 --- Comment #25 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- OK. But DOM UNIMARC authority is the object of an other bug. So I don't think it is needed for this one. M. saby -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8252 --- Comment #27 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- (In reply to Frédéric Demians from comment #26)

I don't like the idea of letting this patch entering into Koha.

I agree, the idea was just to know if the "diagnose" was good. If so, as you say, we must change the koha-indexdefs-to-zebra.xsl, and check it does not break anything. It means

that for UNIMARC biblio-zebra-indexdefs.xsl will not be deduced automatically from biblio-koha-indexdefs.xml, like for MARC21. There is a risk that a library modify it indexes XML config file and overwrite you handcrafted XSL file.

Could you point specifically a portion of your XSL file that deal with coded fields substrings?

I wrote this patch months ago, so I don't remember well the details. I can't answer today, but I will try to investigate but maybe the people who know well koha-indexdefs-to-zebra.xsl (who wrote it first?) could gave an answer quicker than me. Mathieu -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8252 --- Comment #29 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- Galen, Frederic : Hopefully I wrote down the problem in comment 8, because I did not remember it this morning ;-) In MARC21, indexing substrings is needed for controled field (001-009, with no subfields) But in UNIMARC it is needed for subfields of 1XX fields too. etc/zebdradb/xml/koha-indexdefs-to-zebra.xsl does only support indexing a position within a subfield in handle-one-index-control-field and "kohaidx:index_leader" mode="secondary" templates. It is good for MARC21, but not for UNIMARC. We need a template to index position within a subfield for UNIMARC 1XX fields too. Current code for handle-one-index-control-field xsl:template name="handle-one-index-control-field"> 214 <xsl:variable name="offset"><xsl:value-of select="@offset"/></xsl:variable> 215 <xsl:variable name="length"><xsl:value-of select="@length"/></xsl:variable> 216 <xsl:variable name="indexes"> 217 <xsl:call-template name="get-target-indexes"/> 218 </xsl:variable> 219 <z:index> 220 <xsl:attribute name="name"><xsl:value-of select="normalize-space($indexes)"/></xsl:attribute> 221 <xslo:value-of> 222 <xsl:attribute name="select"> 223 <xsl:choose> 224 <xsl:when test="@length"> 225 <xsl:text>substring(., </xsl:text> 226 <xsl:value-of select="$offset + 1" /> 227 <xsl:text>, </xsl:text> 228 <xsl:value-of select="$length"/> 229 <xsl:text>)</xsl:text> 230 </xsl:when> 231 <xsl:otherwise> 232 <xsl:text>.</xsl:text> 233 </xsl:otherwise> 234 </xsl:choose> 235 </xsl:attribute> 236 </xslo:value-of> 237 </z:index> 238 </xsl:template> Mathieu -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8252 --- Comment #31 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- For authorities : 101, 102, 152, 160 indexed in a classical way the other must be indexed with "positional indexing". Mathieu -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8252 --- Comment #33 from Frédéric Demians <frederic@tamil.fr> --- Created attachment 21430 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=21430&action=edit XSL handling subfield substring extraction This patch modify koha-indexdefs-to-zebra.xsl in order to add the ability to populate indexes with subfield substring. It's now possible to understand such construction as: <index_subfields xmlns="http://..." tag="100" subfields="a" offset="7" length="1"> <target_index>tpubdate:s</target_index> </index_subfields> -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8252 --- Comment #35 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- In fact, xml file is correct (same values as in UNIMARC manual, which is also begining by "0" position), and xsl wrong. So I am correcting xsl. It was signed off by Bernardo, but as - according to logic - it should NOT have worked, I will ask for a new sign off. Followup coming. Mathieu -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8252 --- Comment #37 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- I applied all my patches, then yours, and I run xsltproc. I make a diff between the resulting zebra xsl files A lot of changes like this one : - <xslo:value-of select="substring(., 8, 1)"/> + <xslo:value-of select="substring(., 9, 1)"/> A new template called : + <xslo:apply-templates mode="index_all"/> [...] - <xslo:template match="*"> + <xslo:template mode="index_all" match="text()"> So, first point, your patch is working well ;-) For the positions : Ex : pubdate In the manual (which starts counting position from 0) : position 8, length 1 http://multimedia.bnf.fr/unimarcb_trad/B100-6-2010.pdf So I suppose it must be position 8 in koha-xml file, and position 9 in zebra-xsl file. That's what we get. I copied the resulting koha-xsl file into my INSTALLDIR/etc/zebradb/marc_defs/unimarc/biblios/ Then I ran rebuild_zebra.pl -x -b -v -r I did some advanced searches with coded fields, it seems to work. Of course, I won't sign off by myself... So, for the moment, I sign off your patch. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8252 --- Comment #39 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- Created attachment 21438 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=21438&action=edit [PATCH] Bug 8252: Fixing biblio-zebra-indexdefs.xsl This patch fixes biblio-zebra-indexdefs.xsl files. It was generated from biblio-koha-indexdefs.xsm with the new koha-indexdefs-to-zebra.xsl amended by F. Démians's patch. To test : - take a DOM UNIMARC Koha - apply all the patchs of 8252 bug, including this one - copy src/etc/zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl to your etc/zebradb/marc_defs/unimarc/biblios/ located in your installation directory - run rebuid_zebra -b -x -r -v - make advanced searches on staff interface and opac, on coded fields indexes (Audience, Literary genre, Biography, Illustration, Content, Video Types, Serial Type, Periodicity, Regularity, Picture) M. Saby -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8252 --- Comment #40 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- Created attachment 21439 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=21439&action=edit Some UNIMARC records Here are some unimarc records if it could be useful for some people to test the patch. M. Saby -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8252 Frédéric Demians <frederic@tamil.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #21438|0 |1 is obsolete| | --- Comment #42 from Frédéric Demians <frederic@tamil.fr> --- Created attachment 21454 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=21454&action=edit Bug 8252: Fixing biblio-zebra-indexdefs.xsl Ok for me. This patch put in sync indexes XSL definition with authoritative XML definition. Subsequently, it won't be difficult to amend DOM UNIMARC indexes definition if necessary. And, as it is, I don't see any regression, whereas I can see huge improvements. Thanks Mathieu! -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8252 --- Comment #43 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- (In reply to Frédéric Demians from comment #41)

Created attachment 21452 [details] Few UNIMARC biblio records in MARCXML

I replace your sample file, which is in ISO2709, with a MARCXML file containing less records. This way Zebra indexing can be tested directly with:

xsltproc biblio-zebra-indexdefs.xsl bib-unimarc.xml

I did not know it was possible! (By the way, & out of scope, importing xml records into Koha would be a nice enh) Mathieu -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8252 Galen Charlton <gmcharlt@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |rel_3_14_candidate -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8252 Jonathan Druart <jonathan.druart@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #21057|0 |1 is obsolete| | Attachment #21058|0 |1 is obsolete| | Attachment #21059|0 |1 is obsolete| | Attachment #21437|0 |1 is obsolete| | Attachment #21454|0 |1 is obsolete| | --- Comment #46 from Jonathan Druart <jonathan.druart@biblibre.com> --- Created attachment 21945 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=21945&action=edit Bug 8252: Fix indexing of UNIMARC 1xx for GRS-1 Before fixing UNIMARC DOM indexing, we must fix GRS-1 indexing 1) In advanced search, some Coded fields index are not working : Print, Illustration, Content 2) Country-heading index is not working 3) Some subfields are indexed in wrong indexes : 102$a should be in Country-publication instead of Country-heading (non defined in bib1.att) 106$a, filled only for printed works, should be in ff88-23 (form of item) instead of itype (ff88-23 is made for Marc21 008 pos 23, which contains the same data as 106a) 200$b should be in Material-type instead of (or in addition to) itype and itemtype (Material-type :"free-form string, ... that describes the material type of the item, e.g., cassette, kit, computer database, computer file.") 100$a pos 22-24 should not be indexed as "ln" : it is the language of the record, not the language of the ressource 4) Index names are too long : if we index new positions of coded fields, with existing names it breaks zebra indexing (there must be a limit in line lenghth in record.abs?) 5) There are a lot of warns when rebuiding zebra. This patch make some changes in bib1.att (could be used later to improve search) : - fixing wording for att 51 and 1012 - adding comments for attributes based on MARC21 008 field (8800-8841) - creating 8806 (tpubdate), 8838 (Modified-code), 8818 (ff8-18), 8840 (ff8-18-21), 8819 (ff8-19), 8821 (ff8-21), 8828 (ff8-28), 8830 (ff8-30), 8831 (ff8-31) - creating attributes specific to UNIMARC : 9701-9707 (Video-mt, Graphics-type, Graphics-support, Title-page-availability, Cumulative-index-availability, script-Title, char-encoding) - setting apart 3 blocks of attributes, so it could be easy to make further changes : -- common to Marc21 and UNIMARC : 8806, 8822, 8838 -- slightly different in Marc21 and UNIMARC (different meanings according to the type of the record => don't match a single UNIMARC field) -- specific to UNIMARC : 9701-9707 in ccl.properties : - creating a new index: Country-publication 1=1053 - suppressing some warns by mapping with bib1 att: Date-time-last-modified, Name, rtype, Music-number - defining indexes using the 3 blocks attributes defined in bib1 (common to Marc21 and UNIMARC, slightly different, specific to UNIMARC) in record.abs : - renaming some index for 100-105-110 fields - correcting indexing of 102$a (country of publication) 106$a (ff88-23) 100$a pos 22-24 (language of record, no more indexed) 105$a pos. 0-3 (illustration code) 200$b (for the moment, I keep it indexed in itype and itemtype, but also Material-Type) in C4/Search.pm : - adding "Country-publication" index in opac and staff interface template subtypes_unimarc.in : - renaming indexes to take into account the changes made to Zebra config files To test (this cannot be done with a sandbox) : 1) apply the patch in a UNIMARC GRS-1 Koha instance 2) copy the following files from the etc/zebradb of your source directory into the etc/zebradb of your main Koha directory : -- etc/zebradb/biblios/etc/bib1.att -- etc/zebradb/ccl.properties -- etc/zebradb/marc_defs/unimarc/biblios/record.abs 3) reindex your data (rebuild_zebra -x -b -r -v) 4) try to use those Coded fields indexes in Advanced search, in OPAC and Staff interface (available after clicking on "More options", then on "Coded information filters"): Audience, Print, Literary genre, Biography, Illustration, Content, Video Types, Serials, Serial Type, Periodicity, Regularity 5) try to search "Country-publication=FR" in simple search Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> No koha-qa errors. Tests for GRS-1 Followed test plan Search by coded fields works, but only on OPAC, on staff there are few options Search by Country-publication works after patch Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8252 --- Comment #48 from Jonathan Druart <jonathan.druart@biblibre.com> --- Created attachment 21947 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=21947&action=edit Bug 8252 : Followup for Date/time-last-modified and Music number This followup restores the original wording of "Date/time-last-modified" index, and change the name of "Music-number" index to "Number-music-publisher" To test : 1. In a UNIMARC Koha instance 2. apply patchs #1, #2 and this followup 3. copy from src/etc/zebradb directory to the etc/zebradb/ in your main koha directory the following files : -- zebradb/biblios/etc/bib1.att -- zebradb/ccl.properties -- zebradb/marc_defs/unimarc/biblios/record.abs -- zebradb/marc_defs/unimarc/biblios/biblio-koha-indexdefs.xml -- zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl 4. rebuild zebra with -b -x -v -r options 5. write a value like "test071a" in 071$a field in a record 6. check if you can find this record with this search : "ccl=Number-music-publisher:test071a" Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> No koha-qa errors. Test Copy files reindex full Modify a couple of record to add 071a with test message Reindex -v -z -b -x Search test message as described and found modified records. Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8252 --- Comment #50 from Jonathan Druart <jonathan.druart@biblibre.com> --- Created attachment 21949 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=21949&action=edit Bug 8252: Fixing biblio-zebra-indexdefs.xsl Bug 8252: Fixing biblio-zebra-indexdefs.xsl This patch fixes biblio-zebra-indexdefs.xsl files. It was generated from biblio-koha-indexdefs.xsm with the new koha-indexdefs-to-zebra.xsl amended by F. Démians's patch. To test : - take a DOM UNIMARC Koha - apply all the patchs of 8252 bug, including this one - copy src/etc/zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl to your etc/zebradb/marc_defs/unimarc/biblios/ located in your installation directory - run rebuid_zebra -b -x -r -v - make advanced searches on staff interface and opac, on coded fields indexes (Audience, Literary genre, Biography, Illustration, Content, Video Types, Serial Type, Periodicity, Regularity, Picture) Signed-off-by: Frédéric Demians <f.demians@tamil.fr> Ok for me. This patch put in sync indexes XSL definition with authoritative XML definition. Subsequently, it won't be difficult to amend DOM UNIMARC indexes defintion if necessary. And, as it is, I don't see any regression, whereas I can see huge improvements. Thanks Mathieu! Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8252 --- Comment #52 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- yippee! Mathieu -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8252 --- Comment #53 from Paul Poulain <paul.poulain@biblibre.com> --- Hi Mathieu, I've giving a try to this patch, it fixes some of the errors that appears in zebra logs. But the following aren't: 21:12:42-22/10 zebraidx(29295) [warn] Index 'UPC' not found in attset(s) 21:12:42-22/10 zebraidx(29295) [warn] Index 'UPC' not found in attset(s) 21:12:42-22/10 zebraidx(29295) [warn] Index 'Serials' not found in attset(s) 21:12:42-22/10 zebraidx(29295) [warn] Index 'Serials' not found in attset(s) 21:12:42-22/10 zebraidx(29295) [warn] Index 'Printed-music' not found in attset(s) 21:12:42-22/10 zebraidx(29295) [warn] Index 'Printed-music' not found in attset(s) 21:12:42-22/10 zebraidx(29295) [warn] Index 'Electronic-ressource' not found in attset(s) 21:12:42-22/10 zebraidx(29295) [warn] Index 'Title-host' not found in attset(s) 21:12:42-22/10 zebraidx(29295) [warn] Index 'title-host' not found in attset(s) 21:12:42-22/10 zebraidx(29295) [warn] Index 'Title-host' not found in attset(s) 21:12:42-22/10 zebraidx(29295) [warn] Index 'title-host' not found in attset(s) 21:12:42-22/10 zebraidx(29295) [warn] Index 'lcn' not found in attset(s) 21:12:42-22/10 zebraidx(29295) [warn] Index 'lcn' not found in attset(s) (note that title-host should be spelled Title-host in 2 places) Some questions: * is there any reason why you haven't fixed those attributes as well ? * how to fix them ? Add another UNIMARC specific attribute into bib1.att/ccl.properties, change index name in record.abs or remove it ? Thanks for your feedback ! (I may provide a patch, in another bug) -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8252 --- Comment #55 from mathieu saby <mathieu.saby@univ-rennes2.fr> ---

...

* how to fix them ? Add another UNIMARC specific attribute into bib1.att/ccl.properties, change index name in record.abs or remove it ?

I don't know it right know, but I'm pretty certain the same logic could be apply for them : check if these indexes can be mapped with existing ccl/bib1 values. And if not, create custom ccl/bib1 values for doing that. Mathieu -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8252 Paul Poulain <paul.poulain@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |11119 -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8252 --- Comment #58 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- I cannot change the bug's status to "Need sign off", so I will create a new one. M. Saby -- You are receiving this mail because: You are watching all bug changes.