http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 --- Comment #1 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- Created attachment 22733 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=22733&action=edit Improve UNIMARC biblio indexing This patch makes the following changes to UNIMARC biblio indexing : A. Changes to UNIMARC conf files 1. add comments to biblio-koha-indexdefs.xml 2. make biblio-koha-indexdefs.xml more compact by grouping some declarations Ex : 200$f and 200$g => one declaration for 200$fg 3. suppress unneeded declarations (indexing of some 4XX fields and 6XX fields not in unimarc format) 4. unindex some fields unneeded by most users (328, 207,230,210a,215) 6. replace Title-host with Host-item -- see bug 11119 7. index 208 in Material-Type -- see bug 11119 8. index 100 pos 8-9 and 9-12 in pubdate, but not in pubdate:s This change will only index 2 new fields, but not change the current sorting (made on 210$d) More work is needed for improving date management. 9. Index all subfields of note 318 and 327 in note index 10. Index 304 and 327 in title index as well as note index 327 can contain a list of titles included in a work 304 can contain the title of the original work in case of a translation 11. Index 314 in author index as well as note index 314 can contain authors not mentionned in 200$f/g (the 4th, 5th etc author) 12. Index 328 note in Dissertation-information as well as note B. Changes to ccl.properties : 1. add a new index Dissertation-information (1056) 2. fix EAN, pubdate and acqdate (they were not linked with bib1 attributes) C. Changes to Search.pm 1. add Dissertation-information and suppress Title-host and UPC D. Changes to QP config file queryparser.yaml 1. add Dissertation-information 2 fix EAN, pubdate and acqdate Test plan : If you cannot test in GRS1, test only in DOM, as GRS will be deprecated. 1. Apply the patch in a UNIMARC Koha running with DOM and ICU 2. copy src/etc/searchengine/queryparser.yaml into the main config directory of QP 2. copy src/etc/zebradb/* (all the tree) into the main config directory of Zebra 3. reindex biblios (rebuild_zebra.pl -r -b -x -v) 4. test note index : make some searches on 318$b or 327$b 5. test author index : make some searches on 314 field 6. test title index : make some searches on 304 and 327 field 7. test Dissertation-information index : make some searches on 328 field 8. In a record, put in the dates of 100 fields the values "1000" and "1001" ; try to search a book written in year 1000, you should find the record ; idem for year 1001 9. Regression test (with and without QP activated) : make some searches and sort by date. It should work (well or bad...) as before 10. Regression test (with and without QP activated) : make some searches on EAN (073). It should work as before -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 mathieu saby <mathieu.saby@univ-rennes2.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |In Discussion --- Comment #2 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- something went wrong... -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 --- Comment #4 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- (In reply to Zeno Tajoli from comment #3)

So, you prefer to refine the code ?

What do you mean by that? I made a mistake, and forgot to join the zebra xsl file. I'm testing and correcting the patch in a few minutes. And I am considering making a date sort on 100 coded field instead of 210$d. Mathieu -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 --- Comment #6 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- (In reply to Zeno Tajoli from comment #5)

Well, in fact it is strange to see 'In discussion' after only two comment with a patch attached. So I guess that you want to refine the patch for same motivation. Only this. We will try to sign it when you have finish.

Oh, ok. So, yes, refining and fixing ;-) -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 --- Comment #8 from Zeno Tajoli <z.tajoli@cineca.it> --- No, tag 359 is not wide used in Italy. Bye -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 mathieu saby <mathieu.saby@univ-rennes2.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #22733|0 |1 is obsolete| | --- Comment #9 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- Created attachment 22737 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=22737&action=edit Improve UNIMARC biblio indexing This patch makes the following changes to UNIMARC biblio indexing : A. Changes to UNIMARC conf files 1. add comments to biblio-koha-indexdefs.xml 2. make biblio-koha-indexdefs.xml more compact by grouping some declarations Ex : 200$f and 200$g => one declaration for 200$fg 3. suppress unneeded declarations (indexing of some 4XX fields and 6XX fields not in unimarc format) 4. unindex some (sub)fields unneeded by most users (318, 207,230,210a,215, 4XXd) 5. change the way 308 field is indexed (no visible changes) 6. replace Title-host with Host-item -- see bug 11119 7. index 208 in Material-Type -- see bug 11119 8. index 100 pos 8-9 and 9-12 in pubdate:y and pubdate:n 9. index 100 pos 8-9 in pubdate:s instead of 210$d 10. Index all subfields of note 334 and 327 in note index 11. Index 304 and 327 in title index as well as note index 327 can contain a list of titles included in a work 304 can contain the title of the original work in case of a translation 12. Index 314 in author index as well as note index 314 can contain authors not mentionned in 200$f/g (the 4th, 5th etc author) 13. Index 328 note in Dissertation-information as well as note 14. Index 328$t in Title B. Changes to ccl.properties : 1. add a new index Dissertation-information (1056) 2. fix EAN, pubdate and acqdate (they were not linked with bib1 attributes) C. Changes to Search.pm 1. add Dissertation-information and suppress Title-host and UPC D. Changes to QP config file queryparser.yaml 1. add Dissertation-information 2 fix EAN, pubdate and acqdate Test plan : If you cannot test in GRS1, test only in DOM, as GRS will be deprecated. 1. Apply the patch in a UNIMARC Koha running with DOM and ICU 2. copy src/etc/searchengine/queryparser.yaml into the main config directory of QP 3. copy src/etc/zebradb/ccl.properties into the main config directory of Zebra 4. copy src/etc/zebradb/marc_defs/unimarc/biblio/* into the main config directory of Zebra 5. reindex biblios (rebuild_zebra.pl -r -b -x -v) 6. test note index : make some searches on 334$b or 327$b 7. test author index : make some searches on 314 field 8. test title index : make some searches on 304 and 327 field, make a search on 328$t subfield 9. test Dissertation-information index : make some searches on 328 field 10. In a record, put in the dates of 100 fields the values "1000" (1st date) and "1001" (2d date) ; try to search a book written in year 1000, you should find the record ; idem for year 1001 11. make some searches and sort by date. It should work better as before, especially if you have values like "c2009" or "impr. 2010" in 210 field 12. Regression test : make some searches on several indexes, like EAN, etc. It should work as before Test 10-12 with and without Queryparser activated. Of course, to test search and sort by dates, you need to have full records, with dates in 100 field as well as 210 field. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 mathieu saby <mathieu.saby@univ-rennes2.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 --- Comment #12 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- *** Bug 11026 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 --- Comment #14 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- Not in advanced search, but in normal search, try "Dissertation-information:xxxxx" Mathieu -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 Paola Rossi <paola.rossi@cineca.it> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |paola.rossi@cineca.it --- Comment #16 from Paola Rossi <paola.rossi@cineca.it> --- Kind Mathieu Saby, I install master 3.13.00.042 choosing ICU, DOM, unimarc, as required, and the Italian version of koha. A problem appears at 5-th step of the test plan, with this warning messagge: ./rebuild_zebra.pl -r -b -x -v Zebra configuration information ================================ Zebra biblio directory = /home/koha/bug-11202/var/lib/zebradb/biblios Zebra authorities directory = /home/koha/bug-11202/var/lib/zebradb/authorities Koha directory = /var/root-koha/bug-11202 BIBLIONUMBER in : 001$@ BIBLIOITEMNUMBER in : 090$a ================================ skipping authorities ==================== exporting biblio ==================== 1 Records exported: 1 ==================== REINDEXING zebra ==================== 16:35:52-07/11 zebraidx(9494) [warn] /home/koha/bug-11202/etc/zebradb/biblios/etc/dom-config.xml: could not parse xslt stylesheet /home/koha/bug-11202/etc/zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl 16:35:52-07/11 zebraidx(9494) [warn] /home/koha/bug-11202/etc/zebradb/biblios/etc/dom-config.xml: could not parse xslt stylesheet /home/koha/bug-11202/etc/zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl ==================== CLEANING ==================== Then the search answers erroneously zero findings. Can you help me? -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 --- Comment #18 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- thank you for testing. I suppose you don't have only 1 record in your catalog? Could you tell me if italian unimarc has something different from standard ifla unimarc? Mathieu -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 --- Comment #20 from Paola Rossi <paola.rossi@cineca.it> --- (In reply to mathieu saby from comment #19)

[...] if you run rebuild_zebra without applying my patch, do you have some errors too ?

Mathieu

Kind Mathieu, I tested on stable and on master on 2013-11-07, and I had no error in both. Thanks, Paola -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 --- Comment #22 from Paola Rossi <paola.rossi@cineca.it> --- (In reply to mathieu saby from comment #18)

thank you for testing. I suppose you don't have only 1 record in your catalog? Could you tell me if italian unimarc has something different from standard ifla unimarc?

Mathieu

The error occurs indexing only one of these bibliographic records: Nilo, antico Nilo / Folco Quilici 1989 Il ‰Nilo / Pier Domenico Boccardi 1903 Verde Nilo / Cesare Brandi 2001 in target Italian Bibliographic Service (SBN): opac.sbn.it:2100, database nopac These are three standard records. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 --- Comment #24 from Paola Rossi <paola.rossi@cineca.it> ---

Could you run rebuild_zebra on one of this record, with "-v -v" option instead of just "-v", and paste the results here please?

I've just sent it to your personal email. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 --- Comment #26 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- Thanks. indeed, it's looong ;-) I crashed my etc/ dir with a wrong manipulation, so, still a little work to do have an answer... -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 --- Comment #28 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- Created attachment 22973 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=22973&action=edit [PATCH 2] Bug 11202 : Fix biblio-zebra-indexdefs.xsl 2 lines were missing in biblio-zebra-indexdefs.xsl, so Zebra could not rebuild the index. This patch should fix that. Apply this patch with main patch, and follow the test plan of main patch. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 Paola Rossi <paola.rossi@cineca.it> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA --- Comment #29 from Paola Rossi <paola.rossi@cineca.it> --- Thanks, Mathieu. Now Zebra rebuilds the index. Step 10: 1) In a record, I put in the dates of 100 fields the values "1000" (1st date, 9-12) and "1001" (2d date, 13-16), and nothing in 210$d. Searching (in advanced search) year 1000, I don't find the record; neither searching 1001. The answer is zero findings for 'yr,st-numeric: 1000'. Searching "yr:1000" in simple search, I don't find the record. The answer is zero findings for 'yr:1000'. 2) In a record, I put in the dates of 100 fields the values "1000" (1st date, 9-12) and "1001" (2d date, 13-16), and the value "1000" in 210$d. Searching (in advanced search) year 1000, I find the record. Searching "yr:1000" in simple search, I don't find the record. The answer is zero findings for 'yr:1000'. 3) In a record, I put in the dates of 100 fields the values "1903" (1st date, 9-12) and "1904" (2d date, 13-16), and nothing in 210$d. Searching (in advanced search or in simple search) year 1903, I don't find the record (like 1). So I pass the patch to "Failed QA" status. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 --- Comment #31 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- The year index in dropdown list does not function well. The request sent to zebra is broken (in master) I've made a ticket to our vendor for that. For searching by date, the only way is to use the "filters" that appears in advanced search. These filters can be used even if they are no other search criteria (so they are not really filters ;-) I've proposed to our vendor to suppress the year index from dropdown list. It's unusable on UX point of view, and it is broken... We should use the filters. Well, now, I'm getting back to the bug itself ;-) Mathieu -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 --- Comment #33 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- oups, I made some mistakes. Fields 411 and others fields 4xx are missing. please do not test while it is fixed! -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 --- Comment #35 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- To test this bug: apply the 3 patches. Look at test plan of 1st patch and 3rd patch (whitch only adds a few indexes), and comments 30 to 32 if you have trouble for testing the new way of indexing dates. Mathieu -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 mathieu saby <mathieu.saby@univ-rennes2.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #22750|0 |1 is obsolete| | Attachment #22973|0 |1 is obsolete| | Attachment #23041|0 |1 is obsolete| | --- Comment #36 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- Created attachment 23108 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=23108&action=edit [PATCH] Bug 11202 - Improve UNIMARC biblio indexing All previous patches (non signed off) squashed, with little changes Test plan updated This patch makes the following changes to UNIMARC biblio indexing : A. Changes to record.abs and biblio-koha*/biblio-zebra* conf files 1. Changes with no expected functional effects * add comments to biblio-koha-indexdefs.xml * make biblio-koha-indexdefs.xml more compact by grouping some declarations Ex : 200$f and 200$g => one declaration for 200$fg * reorganizes definitions of 5XX fields * suppress unneeded declarations (indexing of some 4XX fields and 6XX fields not in UNIMARC format) * change the way 308 field is indexed 2. Changes with functional effects * index new fields added in UNIMARC format in 2012 to make it more FRBR compliant : 506, 507, 576, 577. These fields are experimental in most UNIMARC countries. They are not currently used in France but will probably be in some months or years. * unindex some (sub)fields unneeded by users (207,230,215,4XXd) * index 208 in Material-Type -- see bug 11119 * index 100 pos 8-9 and 9-12 in pubdate:y and pubdate:n * index 100 pos 8-9 in pubdate:s instead of 210$d * Index all subfields of note 334 and 327 in note index * Index 304 and 327 in title index as well as note index 327 can contain a list of titles included in a work 304 can contain the title of the original work in case of a translation * Index 314 in author index as well as note index 314 can contain authors not mentionned in 200$f/g (the 4th, 5th etc author) * Index 328 note in Dissertation-information as well as note * Index 328$t in Title * replace Title-host with Host-item -- see bug 11119 B. Changes to ccl.properties : 1. add a new index Dissertation-information (1056) 2. fix EAN, pubdate and acqdate (they were not linked with bib1 attributes) C. Changes to Search.pm 1. add Dissertation-information and suppress Title-host and UPC D. Changes to QP config file queryparser.yaml 1. add Dissertation-information 2 fix EAN, pubdate and acqdate Test plan : If you cannot test in GRS1, test only in DOM, as GRS will be deprecated... 1. Prepare Koha : * Apply the patch in a UNIMARC Koha running with DOM and ICU * copy src/etc/searchengine/queryparser.yaml into the main config directory of QP * copy src/etc/zebradb/ccl.properties into the main config directory of Zebra * copy src/etc/zebradb/marc_defs/unimarc/biblio/* into the main config directory of Zebra * reindex biblios (rebuild_zebra.pl -r -b -x -v) 2. test note index : make some searches on 334$b or 327$b 3. test author index : make some searches on 314 field 4. test title index : make some searches on 304 and 327 field, make a search on 328$t subfield 5. test Dissertation-information index : make some searches on 328 field 6. test 506,507,576,577 fields (as they are mostly experimental, and as index definition is trivial, this test can be skipped) 7. In a record, put in the dates of 100 fields the values "1000" (1st date) and "1001" (2d date) ; try to search a book written in year 1000, you should find the record ; idem for year 1001 8. make some searches and sort by date. It should work better as before, especially if you have values like "c2009" or "impr. 2010" in 210 field 9. Regression test : make some searches on several indexes, like EAN, etc. It should work as before Test 2-9 with and without Queryparser activated. Of course, to test search and sort by dates, you need to have full records, with dates in 100 field as well as 210 field. To search by date, you need to use advanced search, click on "More" and use date filter. Do not use "yr" index in simple search, it does not work. Do not use "date" index in dropdown list, it does not work either. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 I'm just a bot <gitbot@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |gitbot@bugs.koha-community. | |org When did the bot| |2013-11-23 last check this| | --- Comment #37 from I'm just a bot <gitbot@bugs.koha-community.org> --- Patch applied cleanly, go forth and signoff -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 mathieu saby <mathieu.saby@univ-rennes2.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #23108|0 |1 is obsolete| | --- Comment #39 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- Created attachment 23238 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=23238&action=edit Bug 11202 - Improve UNIMARC biblio indexing Sorry :( It's fixed. Mathieu -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 Paola Rossi <paola.rossi@cineca.it> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA --- Comment #40 from Paola Rossi <paola.rossi@cineca.it> --- Thanks, Mathieu. Now rebuilding zebra indexes is OK. I turn "UseQueryParser" preference to the second option "Try". As required at step 5, I submit: Dissertation-information: XXXX in homepage, selecting tab "Search the catalog". Koha answers: Home › Catalogo › Ricerca di 'pqf=@and @and @or @or @attr 1=1016 @attr 5=1 @attr 4=6 "Dissertation-information:" @attr 9=20 @attr 2=102 @attr 5=1 @attr 4=6 "Dissertation-information:" @attr 9=34 @attr 2=102 @attr 5=1 @attr 4=6 "Dissertation-information:" @or @or @attr 1=1016 @attr 5=1 @attr 5=1 @attr 4=6 "XXXX" @attr 9=20 @attr 2=102 @attr 5=1 @attr 5=1 @attr 4=6 "XXXX" @attr 9=34 @attr 2=102 @attr 5=1 @attr 5=1 @attr 4=6 "XXXX" @or @or @attr 1=1016 @attr 5=1 @attr 5=1 @attr 5=1 @attr 4=6 "#relevance_asc" @attr 9=20 @attr 2=102 @attr 5=1 @attr 5=1 @attr 5=1 @attr 4=6 "#relevance_asc" @attr 9=34 @attr 2=102 @attr 5=1 @attr 5=1 @attr 5=1 @attr 4=6 "#relevance_asc"' Nessun risultato trovato Nessun risultato rispondente alla tua ricerca per 'pqf=@and @and @or @or @attr 1=1016 @attr 5=1 @attr 4=6 "Dissertation-information:" @attr 9=20 @attr 2=102 @attr 5=1 @attr 4=6 "Dissertation-information:" @attr 9=34 @attr 2=102 @attr 5=1 @attr 4=6 "Dissertation-information:" @or @or @attr 1=1016 @attr 5=1 @attr 5=1 @attr 4=6 "XXXX" @attr 9=20 @attr 2=102 @attr 5=1 @attr 5=1 @attr 4=6 "XXXX" @attr 9=34 @attr 2=102 @attr 5=1 @attr 5=1 @attr 4=6 "XXXX" @or @or @attr 1=1016 @attr 5=1 @attr 5=1 @attr 5=1 @attr 4=6 "#relevance_asc" @attr 9=20 @attr 2=102 @attr 5=1 @attr 5=1 @attr 5=1 @attr 4=6 "#relevance_asc" @attr 9=34 @attr 2=102 @attr 5=1 @attr 5=1 @attr 5=1 @attr 4=6 "#relevance_asc"'. So I pass the patch to "Failed QA" status. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 mathieu saby <mathieu.saby@univ-rennes2.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff --- Comment #42 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- Hello Paola It's a bug of the queryparser. I've created a new bug: http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11353 So, you must use only lowercases in indexes : "dissertation-information" and NOT "Dissertation-information". I update test plan. Mathieu -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 Paola Rossi <paola.rossi@cineca.it> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #23348|0 |1 is obsolete| | --- Comment #44 from Paola Rossi <paola.rossi@cineca.it> --- Created attachment 23579 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=23579&action=edit Improve UNIMARC biblio indexing I test the patch against master 3.15.00.004. Simple sort is ok when default sort is not set to relevance-asc (see bug 11255). I sign off the patch. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 --- Comment #45 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- Thank you Paola! -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |katrin.fischer@bsz-bw.de --- Comment #47 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I am quite torn about this patch. I feel it's doing way too many things in one single change. It's really hard to test that way and I am not super confident about doing it, but going to give it a try. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 --- Comment #49 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I will have some comments soon. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 --- Comment #51 from Jared Camins-Esakov <jcamins@cpbibliography.com> --- (In reply to Katrin Fischer from comment #50)

I am having problems testing pubdate without QueryParser. For the QueryParser pubdate(xxxx) seems to work alright. But pubdate= or pubdate: doesn't seem to work. Any ideas about that?

Yes, dates are searched with parentheses not = or : with QP. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Failed QA --- Comment #53 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- What I did after applying the patch: cp etc/searchengine/queryparser.yaml ../koha-devU/etc/searchengine/queryparser.yaml cp etc/zebradb/ccl.properties ../koha-devU/etc/zebradb/ccl.properties cp etc/zebradb/marc_defs/unimarc/biblios/biblio-koha-indexdefs.xml ../koha-devU/etc/zebradb/marc_defs/unimarc/biblios/biblio-koha-indexdefs.xml cp etc/zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl ../koha-devU/etc/zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl cp etc/zebradb/marc_defs/unimarc/biblios/record.abs ../koha-devU/etc/zebradb/marc_defs/unimarc/biblios/record.abs Completely reindexed the bibliographic data using -b -v -r -x Full reindex and searching for Dissertation-information worked nicely in DOM and GRS1. But... 1) In general I am having problems with pubdate and acqdate searches in UNIMARC compared to their MARC21 equivalents Mathieu, could you please take a look? Syntaxes I tried: pubdate=yyyy, pubdate:yyyy With QueryParser: pubdate(yyyy), acqdate(yyyy) 2) It would be nice if Dissertation-information could also be added for MARC21 on a seaparate bug later on. 3) In general I am a bit worried about removing indexes. I can see, why some subfields are not making much sense to index, I also have read the comments on bug 11119, but some of the changes might be debatable. For example the following change could break functionality some libraries rely on. Maybe it would not hurt to keep those around? -#FIXME Fields 603, 630, 631, 632, 633, 634, 635, 636, 646 are not defined in Unimarc, but may be used by some libraries. -melm 603 Subject,Subject:p -melm 630 Subject,Subject:p -melm 631 Subject,Subject:p -melm 632 Subject,Subject:p -melm 633 Subject,Subject:p -melm 634 Subject,Subject:p -melm 635 Subject,Subject:p -melm 636 Subject,Subject:p -melm 646 Subject,Subject:p Failing for now until 1) can be resolved. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 --- Comment #55 from mathieu saby <mathsabypro@gmail.com> --- I made some tests. Without my patch: 1. with QP enable pubdate(1925) is working pubdate:1925 is working 2. with QP disable pubdate(1925) is not working pubdate:1925 is not working With my patch: Exactly the same behavior So, for me, the patch is correct, regarding pubdate index. It is not possible to build a query with pubdate without QP, but it was still the case in master. Mathieu -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 --- Comment #57 from mathieu saby <mathsabypro@gmail.com> --- I was always said not to use pubdate index like that, so the issue is old, but maybe unimarc specific. It is working only when you use a date in the advanced search filter see comment 30 By the way: If you or Galen ask it, I can try to split the patch, with non functional changes (comments...) in a separate patch. But I prefer not to, if it is not required, because - I could do a mistake and break the patch, - I won't have time to work on Koha this week. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 --- Comment #59 from mathieu saby <mathsabypro@gmail.com> --- Bug 11585 created -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 Jonathan Druart <jonathan.druart@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@biblibre.co | |m --- Comment #60 from Jonathan Druart <jonathan.druart@biblibre.com> --- All looks good to me. I had a discussion with Paul and the only think we found is that the Dissertation-information is not defined for MARC21 (but the index is defined in the ccl.properties). See bug 11620. Marked as Passed QA. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 Jonathan Druart <jonathan.druart@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #23830|0 |1 is obsolete| | --- Comment #61 from Jonathan Druart <jonathan.druart@biblibre.com> --- Created attachment 24751 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=24751&action=edit Bug 11202 - Improve UNIMARC biblio indexing This patch makes the following changes to UNIMARC biblio indexing : A. Changes to UNIMARC conf files 1. add comments to biblio-koha-indexdefs.xml 2. make biblio-koha-indexdefs.xml more compact by grouping some declarations Ex : 200$f and 200$g => one declaration for 200$fg 3. suppress unneeded declarations (indexing of some 4XX fields and 6XX fields not in unimarc format) 4. unindex some (sub)fields unneeded by most users (318, 207,230,210a,215, 4XXd) 5. change the way 308 field is indexed (no visible changes) 6. replace Title-host with Host-item -- see bug 11119 7. index 208 in Material-Type -- see bug 11119 8. index 100 pos 8-9 and 9-12 in pubdate:y and pubdate:n 9. index 100 pos 8-9 in pubdate:s instead of 210$d 10. Index all subfields of note 334 and 327 in note index 11. Index 304 and 327 in title index as well as note index 327 can contain a list of titles included in a work 304 can contain the title of the original work in case of a translation 12. Index 314 in author index as well as note index 314 can contain authors not mentionned in 200$f/g (the 4th, 5th etc author) 13. Index 328 note in Dissertation-information as well as note 14. Index 328$t in Title B. Changes to ccl.properties : 1. add a new index Dissertation-information (1056) 2. fix EAN, pubdate and acqdate (they were not linked with bib1 attributes) C. Changes to Search.pm 1. add Dissertation-information and suppress Title-host and UPC D. Changes to QP config file queryparser.yaml 1. add Dissertation-information 2 fix EAN, pubdate and acqdate Test plan : If you cannot test in GRS1, test only in DOM, as GRS will be deprecated. 1. Apply the patch in a UNIMARC Koha running with DOM and ICU 2. copy src/etc/searchengine/queryparser.yaml into the main config directory of QP 3. copy src/etc/zebradb/ccl.properties into the main config directory of Zebra 4. copy src/etc/zebradb/marc_defs/unimarc/biblio/* into the main config directory of Zebra 5. reindex biblios (rebuild_zebra.pl -r -b -x -v) 6. test note index : make some searches on 334$b or 327$b 7. test author index : make some searches on 314 field 8. test title index : make some searches on 304 and 327 field, make a search on 328$t subfield 9. test dissertation-information index : make some searches on 328 field 10. In a record, put in the dates of 100 fields the values "1000" (1st date) and "1001" (2d date) ; try to search a book written in year 1000, you should find the record ; idem for year 1001 11. make some searches and sort by date. It should work better as before, especially if you have values like "c2009" or "impr. 2010" in 210 field 12. Regression test : make some searches on several indexes, like EAN, etc. It should work as before Test 10-12 with and without Queryparser activated. Be careful: with Queryparser activated, the index names (title, dissertation-information...) must be entered in lowercase only. Of course, to test search and sort by dates, you need to have full records, with dates in 100 field as well as 210 field. Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 Galen Charlton <gmcharlt@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |gmcharlt@gmail.com --- Comment #63 from Galen Charlton <gmcharlt@gmail.com> ---

From my review of this patch, five tests in t/QueryParser.t fail. At first glance, I think the problem is with the tests, though.

-- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11202 Galen Charlton <gmcharlt@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to Master --- Comment #65 from Galen Charlton <gmcharlt@gmail.com> --- Pushed to master. Thanks, Mathieu! -- You are receiving this mail because: You are watching all bug changes.