[Bug 39287] New: Improve "Show volume" search with a more specialized search index
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39287 Bug ID: 39287 Summary: Improve "Show volume" search with a more specialized search index Change sponsored?: --- Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Cataloging Assignee: koha-bugs@lists.koha-community.org Reporter: katrin.fischer@bsz-bw.de QA Contact: testopia@bugs.koha-community.org CC: m.de.rooy@rijksmuseum.nl, michaela.sieber@kit.edu Currently the rcn index used to search for volumes contains all $w subfields from 7xx and 8xx linking fields. In some cases this is too broad, even tho we are additionally limiting to monographs. Example: Search volumes will also find - records with a 775$w - other edition - records with a matching 780/785$w - preceeding or succeeding ... A specific index for searching volumes (rcn-volumes) should contain: - 773$w (we still need to limit to monographic, because these could also be analytcal links) - 800$w - 810$w - 811$w - 830$w Once the index is defined we can update the XSLT and the method for determining if the "Show volumes" link should be shown to use it and have a correct result list. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39287 --- Comment #1 from Jan Kissig <bibliothek@th-wildau.de> --- Created attachment 182765 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=182765&action=edit Bug 39287: Improve "Show volume" search with a more specialized search index The former implementation of "show volume" is relying on an index called rcn which is fed with every 7xx$w and 8xx$w subfield and therefore also shows predecessors and successors to the title. This patch adds a new index only consisting of 773, 800, 810, 811 and 830 $w subfields called rcn-volumes. This index is then used to display the "Show volumes" button and used to limit the results to volumes only. Test plan: 0) set system preference UseControlNumber to Use a) import bug_39287.xml from the attachements f.e. via More -> Cataloging -> Stage records for import Format is MARCXML b) After import search for "main" and open the record with "Main title" c) Click on Show volumes d) recognize search string is "rcn:cnmain NOT (bib-level:a OR bib-level:b)" and 4 results are found e) apply patch f) copy the necessary files to the zebra dirctory sudo cp -i etc/zebradb/biblios/etc/bib1.att /etc/koha/zebradb/biblios/etc/bib1.att sudo cp -i etc/zebradb/ccl.properties /etc/koha/zebradb/ccl.properties sudo cp -i etc/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl /etc/koha/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl g) restart zebra: sudo koha-zebra --restart kohadev h) rebuild the index: sudo koha-rebuild-zebra -b -f -v kohadev i) restart_all j) redo step b) to d) k) recognize search string is now 'rcn-volumes:cnmain NOT (bib-level:a OR bib-level:b)' and only 2 records (volumes) are found l) sign off -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39287 Jan Kissig <bibliothek@th-wildau.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bibliothek@th-wildau.de --- Comment #2 from Jan Kissig <bibliothek@th-wildau.de> --- Created attachment 182766 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=182766&action=edit Record with volumes, predeccessor and successor -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39287 Jan Kissig <bibliothek@th-wildau.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff Assignee|koha-bugs@lists.koha-commun |bibliothek@th-wildau.de |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39287 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |anke.bruns@gwdg.de -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39287 Anke Bruns <anke.bruns@gwdg.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |In Discussion --- Comment #3 from Anke Bruns <anke.bruns@gwdg.de> --- Mit Zebra funktioniert der Patch, jedoch nicht, wenn die Instanz Elasticsearch nutzt. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39287 Anke Bruns <anke.bruns@gwdg.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Discussion |Failed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39287 Anke Bruns <anke.bruns@gwdg.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #182765|0 |1 is obsolete| | --- Comment #4 from Anke Bruns <anke.bruns@gwdg.de> --- Created attachment 187584 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=187584&action=edit Bug 39287: Improve "Show volume" search with a more specialized search index The former implementation of "show volume" is relying on an index called rcn which is fed with every 7xx$w and 8xx$w subfield and therefore also shows predecessors and successors to the title. This patch adds a new index only consisting of 773, 800, 810, 811 and 830 $w subfields called rcn-volumes. This index is then used to display the "Show volumes" button and used to limit the results to volumes only. Test plan: 0) set system preference UseControlNumber to Use a) import bug_39287.xml from the attachements f.e. via More -> Cataloging -> Stage records for import Format is MARCXML b) After import search for "main" and open the record with "Main title" c) Click on Show volumes d) recognize search string is "rcn:cnmain NOT (bib-level:a OR bib-level:b)" and 4 results are found e) apply patch f) copy the necessary files to the zebra dirctory sudo cp -i etc/zebradb/biblios/etc/bib1.att /etc/koha/zebradb/biblios/etc/bib1.att sudo cp -i etc/zebradb/ccl.properties /etc/koha/zebradb/ccl.properties sudo cp -i etc/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl /etc/koha/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl g) restart zebra: sudo koha-zebra --restart kohadev h) rebuild the index: sudo koha-rebuild-zebra -b -f -v kohadev i) restart_all j) redo step b) to d) k) recognize search string is now 'rcn-volumes:cnmain NOT (bib-level:a OR bib-level:b)' and only 2 records (volumes) are found l) sign off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39287 Anke Bruns <anke.bruns@gwdg.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39287 --- Comment #5 from Anke Bruns <anke.bruns@gwdg.de> --- Created attachment 187588 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=187588&action=edit Test plan part 1: Zebra: 0) start instance with searchengine set to Zebra a) set system preference UseControlNumber to Use b) import bug_39287.xml from the attachements f.e. via More -> Cataloging -> Stage records for import Format is MARCXML c) After Test plan part 2: Elasticsearch 0) start instance with searchengine set to Elasticsearch a) Rebuild Elasticsearch index with --reset parameter to apply new mappings. Redo test plan from part 1 step c) to d) and l) m) sign off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39287 --- Comment #6 from Anke Bruns <anke.bruns@gwdg.de> --- Sollten noch weitere Felder außer 773$w und 830$w für Show volumes abgefragt werden? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39287 --- Comment #7 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Anke Bruns from comment #6)
Sollten noch weitere Felder außer 773$w und 830$w für Show volumes abgefragt werden?
yes, the list of fields is in the first comment: - 773$w (we still need to limit to monographic, because these could also be analytcal links) - 800$w - 810$w - 811$w - 830$w -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39287 --- Comment #8 from Anke Bruns <anke.bruns@gwdg.de> --- If the index rcn-volumes is used for the linking between parent and child records in multilevel records, why do we need the resp. w subfields of 800 - Series Added Entry - Personal Name (R) Full | Concise 810 - Series Added Entry - Corporate Name (R) Full | Concise 811 - Series Added Entry - Meeting Name (R) Full | Concise (copied from LOC MARC description). Sorry for asking a potentially dumb question :-D -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39287 --- Comment #9 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- The linking between set and volumes is using different fields depending on if the volumes have dependent or independent titles. One uses 773, the other a combination of 8xx and 490. In the latter case the 8xx has the $w subfield we need to find them. So we can leave out 490, but we need all the other fields. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39287 --- Comment #10 from Anke Bruns <anke.bruns@gwdg.de> --- I wasn't aware, thanks for clarifying. So 800w, 810w and 811w should be added to the attachment; I'll see to it as soon as I manage to recover my testing environment :-D -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39287 --- Comment #11 from Anke Bruns <anke.bruns@gwdg.de> --- (In reply to Anke Bruns from comment #10)
I wasn't aware, thanks for clarifying.
So 800w, 810w and 811w should be added to the attachment; I'll see to it as soon as I manage to recover my testing environment :-D
... I thought it was only in 830$w that a volume links to the parent record. So the link to the parent can also be in 800$w, 810$w or 811$w? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39287 --- Comment #12 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- It could be, depending on the the main heading of the linked record is, if I understand correctly. But it could be that in practice it's not used that much. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39287 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |34482 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34482 [Bug 34482] Improve MARC21 7xx linking fields in bibliographic records [Omnibus] -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39287 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |35330 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35330 [Bug 35330] Split rcn index to allow for more specific searches -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39287 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |42027 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42027 [Bug 42027] Implement "Show volumes" for serial records -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39287 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=35117 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39287 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |44064 Blocks|34482 | CC| |martin.renvoize@openfifth.c | |o.uk Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34482 [Bug 34482] Improve MARC21 7xx linking fields in bibliographic records [Omnibus] https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=44064 [Bug 44064] Match linking $w on the (OrgCode)control number form when UseControlNumber is enabled -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39287 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #187584|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39287 --- Comment #13 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 211469 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=211469&action=edit Bug 39287: Improve "Show volume" search with a more specialized search index The former implementation of "show volume" is relying on an index called rcn which is fed with every 7xx$w and 8xx$w subfield and therefore also shows predecessors and successors to the title. This patch adds a new index only consisting of 773, 800, 810, 811 and 830 $w subfields called rcn-volumes. This index is then used to display the "Show volumes" button and used to limit the results to volumes only. Test plan: 0) set system preference UseControlNumber to Use a) import bug_39287.xml from the attachements f.e. via More -> Cataloging -> Stage records for import Format is MARCXML b) After import search for "main" and open the record with "Main title" c) Click on Show volumes d) recognize search string is "rcn:cnmain NOT (bib-level:a OR bib-level:b)" and 4 results are found e) apply patch f) copy the necessary files to the zebra dirctory sudo cp -i etc/zebradb/biblios/etc/bib1.att /etc/koha/zebradb/biblios/etc/bib1.att sudo cp -i etc/zebradb/ccl.properties /etc/koha/zebradb/ccl.properties sudo cp -i etc/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl /etc/koha/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl g) restart zebra: sudo koha-zebra --restart kohadev h) rebuild the index: sudo koha-rebuild-zebra -b -f -v kohadev i) restart_all j) redo step b) to d) k) recognize search string is now 'rcn-volumes:cnmain NOT (bib-level:a OR bib-level:b)' and only 2 records (volumes) are found l) sign off Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39287 --- Comment #14 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 211470 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=211470&action=edit Bug 39287: (follow-up) Add the rcn-volumes Elasticsearch mapping The previous patch adds the rcn-volumes index to Zebra. Add the same index to the Elasticsearch mappings. To test, with Elasticsearch: 1. Start the instance with Elasticsearch as the search engine 2. Set UseControlNumber to Use 3. Import bug_39287.xml (attached to the bug) from Cataloging > Stage records for import, format MARCXML 4. Rebuild the Elasticsearch index with --reset so that the new mapping is applied 5. Search for "main", open the record "Main title" and click Show volumes => The search is rcn-volumes:cnmain NOT (bib-level:a OR bib-level:b) and only the 2 volumes are found 6. Sign off Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39287 --- Comment #15 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 211471 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=211471&action=edit Bug 39287: (QA follow-up) Complete the rcn-volumes index The rcn-volumes index was only partly added: * The Elasticsearch mapping only had 773$w and 830$w, so volumes linked through 800, 810 or 811 were found with Zebra but not with Elasticsearch. Map the same fields as Zebra, with the settings the other mappings have. * The OPAC "Show volumes" link still searched the rcn index, while whether to show the link is decided with get_marc_volumes, which uses rcn-volumes. Use rcn-volumes there too, as the staff interface does. * Add rcn-volumes to getIndexes in C4::Search, so that a query using it is recognised as a CCL query with Zebra. * Update the get_volumes_query tests for the new index, and add a test that checks that Zebra and Elasticsearch index the same fields for rcn-volumes, and that ccl.properties and bib1.att agree. * Correct the POD of get_marc_volumes. To test: 1. Set UseControlNumber to Use 2. Create a set record (Leader/19 'a', 001 S1) and volumes of it that link with 773$w, 800$w, 810$w, 811$w and 830$w S1, and records that link with 775$w, 776$w, 780$w and 785$w S1 3. Rebuild the search index (with Elasticsearch, with --reset) With Zebra, copy etc/zebradb/biblios/etc/bib1.att, etc/zebradb/ccl.properties and etc/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl to your Zebra configuration first 4. In the staff interface, open the set record and click Show volumes => Only the volumes are found 5. In the OPAC, open the set record and click Show volumes => The same records are found => The link searches rcn-volumes 6. Run $ prove t/Koha_SearchEngine_rcn_volumes.t t/db_dependent/Koha/Biblio.t => SUCCESS: Tests pass! 7. Sign off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39287 --- Comment #16 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 211472 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=211472&action=edit Bug 39287: (QA follow-up) Add the rcn-volumes search field to existing installations The Elasticsearch mappings are stored in the database, and mappings.yaml is only read when the mappings are reset. An existing installation would not get the new rcn-volumes field, so "Show volumes" would find nothing with Elasticsearch until the mappings had been reset. Add the search field and its mappings in an atomic update, as was done for other fields. It also completes the mappings of an installation that only has some of them. To test: 1. Make sure that you have Elasticsearch, and remove the rcn-volumes search field and its mappings if you have them: DELETE FROM search_field WHERE name = 'rcn-volumes'; 2. Run $ perl installer/data/mysql/updatedatabase.pl => The field and its 5 mappings are added 3. Run it again => Nothing is added twice 4. Rebuild the Elasticsearch index and follow the test plan of the previous patch 5. Sign off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39287 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off --- Comment #17 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Hi Jan, Anke, Katrin, I have picked this up, as part of the work on how volumes and components are found with UseControlNumber (bug 34482). I have re-attached your two patches with my sign-off (the subject of Anke's patch, which was the pasted test plan, is now "Bug 39287: (follow-up) Add the rcn-volumes Elasticsearch mapping"), and attached two QA follow-ups on top of them. The Zebra part works well. With a set and volumes linked through 773, 800, 810 and 830, the old rcn search also found records linked as other edition, other format, predecessor and successor (8 records), and rcn-volumes finds only the 4 volumes. What I found and fixed: - The Elasticsearch mapping only had 773 and 830, so volumes linked through 800, 810 or 811 were missing with Elasticsearch only. It now maps the same fields as Zebra, and a test checks that the two stay in line. - The OPAC Show volumes link still searched rcn, while whether to show it is decided with the new index, so I changed it too. - rcn-volumes is added to getIndexes in C4::Search so that it is recognised as a CCL query with Zebra, as was done for cni. - t/db_dependent/Koha/Biblio.t failed because of the new index, so I updated it. - Existing Elasticsearch installations only get new fields when the mappings are reset, so there is a new atomic update, as for other fields. Zebra installations need to copy bib1.att, ccl.properties and biblio-zebra-indexdefs.xsl to their configuration and reindex, and Elasticsearch installations need a reindex, so this needs a release note. I checked it on Zebra and on Elasticsearch 8 with a set record, volumes linked through 773, 800, 810 and 830, and records linked through 775, 776, 780 and 785: both engines now return the same 4 volumes. Tests and the QA script pass. I am setting this to Signed Off so that it can go to QA. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39287 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #187588|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon@bugs.koha-community.org