[Bug 37871] New: FindDuplicate finds weird duplicates
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37871 Bug ID: 37871 Summary: FindDuplicate finds weird duplicates Change sponsored?: --- Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Cataloging Assignee: koha-bugs@lists.koha-community.org Reporter: caroline.cyr-la-rose@inlibro.com QA Contact: testopia@bugs.koha-community.org CC: m.de.rooy@rijksmuseum.nl This keeps happening in my KTD with es7. I create a record with the title "Title", no ISBN, no author, and Koha thinks it's a duplicate of Giulio Cesare. At least it's consistent in finding the wrong record... I use MARC21. To replicate: 1. Open KTD with es7 (`ktd --es7 up`) 2. Go to Cataloging > New record 3. In tab 0, fill out the mandatory fields - 000 - 003 - 005 - 008 - 040c (I put "a" here, but I don't think this matters) 4. In tab 2, fill out the title with the following information (Yes, I just enter "Title" in the title field, and "Remainder of title" in the remainder of title fields, etc.) - 245a: Title - 245n: Number of part - 245p: Title of part - 245b: Remainder of title 5. In tab 9, choose an item type (I chose Books, but I don't think this matters either) 6. Click Save --> Koha asks if this is a duplicate of "Giulio Cesare" -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37871 --- Comment #1 from Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com> --- Created attachment 171219 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=171219&action=edit Screenshot Screenshot of my record with the title field showing (Title, Number of part, Name of part, Remainder of title) and the "Duplicate record suspected" warning message at the top saying this is a possible duplicate of Giulio Cesare. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37871 Phil Ringnalda <phil@chetcolibrary.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |phil@chetcolibrary.org --- Comment #2 from Phil Ringnalda <phil@chetcolibrary.org> --- Why? That record has 246 $iOther title on container:$aGiulio Cesare in Egitto, and Koha indexes 246$i (and for that matter $h, or any imaginary subfield you add to 246 in a framework) in the title index. The "fun" thing about ElasticSearch is you can fix that: Administration - Search engine configuration (Elasticsearch) - Bibliographic records tab, find the title line for 246 and change it from just 246 to 246ab and koha-elasticsearch --rebuild -d -b kohadev. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37871 Jonathan Druart <jonathan.druart@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@gmail.com See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=15248 -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37871 --- Comment #3 from Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com> --- Thanks Phil! I feel like 246$i should not be indexed in the title, is it just me? We could change the default mapping to 246abnp. Opinions? -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37871 --- Comment #4 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Caroline Cyr La Rose from comment #3)
Thanks Phil!
I feel like 246$i should not be indexed in the title, is it just me? We could change the default mapping to 246abnp.
Opinions?
We have had libraries complaining about the title index being too wide in the past. I am not sure if we can make everyone happy here, but maybe removing the $i by default from 246 would be a good start. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37871 Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37871 --- Comment #5 from Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com> --- Created attachment 171488 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=171488&action=edit Bug 37871: Remove extraneous 246 subfields from the title mappings (Elasticsearch, MARC21) This patch limits indexing of field 246 to $a, $b, $n, and $p in various title indexes. Previously, all 246 subfields were indexed, including non-title subfields such as $i (Display text), $g (Miscellaneous information), and linking subfields, making the title index very large and giving false results. To test: 1. Add all the 246 subfields to the default bibliographic framework 1.1. Go to Administration > MARC bibliographic framework 1.2. Click Actions > MARC structure next to the Default framework 1.3. Search for 246 1.4. Click Actions > Edit subfields 1.5. For each subfield, make sure Editor is checked 1.6. Click Save changes 2. Create a new record and fill out all the 246 subfields 2.1. Go to Cataloging 2.2. Click New record 2.3. Fill out the mandatory fields (000, 003, 005, 008, 040$c, 245$a, 942$c) 2.4. Fill out all the subfields in 246 (I simply write the name of the subfield in the text field e.g. Display text in 246$i) 2.5. Click Save 2.6. Click Normal to access the detailed record 3. View the Elastic search record --> All the subfields (including "Display text", "Miscellaneous information" and other non-title subfields) should be indexed in - title - title__suggestion - title-abbreviated - title-abbreviated__sort - title expanded - title-expanded__sort - title-former - title-former__sort 4. Make a title or keyword search for "Display" (or whatever you wrote in 246$i) --> Your record should be in the results 5. Apply patch 6. Rebuild the index using -r koha-elasticsearch --rebuild -d -r -v kohadev 7. Redo step 3 --> Only 246 $a, $b, $n, and $p should be indexed 8. Redo step 4 --> Your record should NOT be in the results -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37871 Phil Ringnalda <phil@chetcolibrary.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37871 Phil Ringnalda <phil@chetcolibrary.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #171488|0 |1 is obsolete| | --- Comment #6 from Phil Ringnalda <phil@chetcolibrary.org> --- Created attachment 171497 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=171497&action=edit Bug 37871: Remove extraneous 246 subfields from the title mappings (Elasticsearch, MARC21) This patch limits indexing of field 246 to $a, $b, $n, and $p in various title indexes. Previously, all 246 subfields were indexed, including non-title subfields such as $i (Display text), $g (Miscellaneous information), and linking subfields, making the title index very large and giving false results. To test: 1. Add all the 246 subfields to the default bibliographic framework 1.1. Go to Administration > MARC bibliographic framework 1.2. Click Actions > MARC structure next to the Default framework 1.3. Search for 246 1.4. Click Actions > Edit subfields 1.5. For each subfield, make sure Editor is checked 1.6. Click Save changes 2. Create a new record and fill out all the 246 subfields 2.1. Go to Cataloging 2.2. Click New record 2.3. Fill out the mandatory fields (000, 003, 005, 008, 040$c, 245$a, 942$c) 2.4. Fill out all the subfields in 246 (I simply write the name of the subfield in the text field e.g. Display text in 246$i) 2.5. Click Save 2.6. Click Normal to access the detailed record 3. View the Elastic search record --> All the subfields (including "Display text", "Miscellaneous information" and other non-title subfields) should be indexed in - title - title__suggestion - title-abbreviated - title-abbreviated__sort - title expanded - title-expanded__sort - title-former - title-former__sort 4. Make a title or keyword search for "Display" (or whatever you wrote in 246$i) --> Your record should be in the results 5. Apply patch 6. Rebuild the index using -r koha-elasticsearch --rebuild -d -r -v kohadev 7. Redo step 3 --> Only 246 $a, $b, $n, and $p should be indexed 8. Redo step 4 --> Your record should NOT be in the results Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org> -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37871 Phil Ringnalda <phil@chetcolibrary.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |caroline.cyr-la-rose@inlibr |ity.org |o.com -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37871 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37871 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #171497|0 |1 is obsolete| | --- Comment #7 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 172165 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=172165&action=edit Bug 37871: Remove extraneous 246 subfields from the title mappings (Elasticsearch, MARC21) This patch limits indexing of field 246 to $a, $b, $n, and $p in various title indexes. Previously, all 246 subfields were indexed, including non-title subfields such as $i (Display text), $g (Miscellaneous information), and linking subfields, making the title index very large and giving false results. To test: 1. Add all the 246 subfields to the default bibliographic framework 1.1. Go to Administration > MARC bibliographic framework 1.2. Click Actions > MARC structure next to the Default framework 1.3. Search for 246 1.4. Click Actions > Edit subfields 1.5. For each subfield, make sure Editor is checked 1.6. Click Save changes 2. Create a new record and fill out all the 246 subfields 2.1. Go to Cataloging 2.2. Click New record 2.3. Fill out the mandatory fields (000, 003, 005, 008, 040$c, 245$a, 942$c) 2.4. Fill out all the subfields in 246 (I simply write the name of the subfield in the text field e.g. Display text in 246$i) 2.5. Click Save 2.6. Click Normal to access the detailed record 3. View the Elastic search record --> All the subfields (including "Display text", "Miscellaneous information" and other non-title subfields) should be indexed in - title - title__suggestion - title-abbreviated - title-abbreviated__sort - title expanded - title-expanded__sort - title-former - title-former__sort 4. Make a title or keyword search for "Display" (or whatever you wrote in 246$i) --> Your record should be in the results 5. Apply patch 6. Rebuild the index using -r koha-elasticsearch --rebuild -d -r -v kohadev 7. Redo step 3 --> Only 246 $a, $b, $n, and $p should be indexed 8. Redo step 4 --> Your record should NOT be in the results Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37871 Martin Renvoize (ashimema) <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |martin.renvoize@ptfs-europe | |.com QA Contact|testopia@bugs.koha-communit |nick@bywatersolutions.com |y.org | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37871 Martin Renvoize (ashimema) <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to main Version(s)| |24.11.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37871 --- Comment #8 from Martin Renvoize (ashimema) <martin.renvoize@ptfs-europe.com> --- Thanks for all the hard work! Pushed to main for the next 24.11.00 release as RM Assistant -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37871 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |additional_work_needed, | |release-notes-needed --- Comment #9 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I am all OK with the change and the push, but I feel the bug description no longer really matches what we did here and maybe a short explanation in release notes would be good as well. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37871 Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|FindDuplicate finds weird |Remove extraneous 246 |duplicates |subfields from the title | |mappings (Elasticsearch, | |MARC21) Text to go in the| |This patch limits indexing release notes| |of field 246 to $a, $b, $n, | |and $p in various title | |indexes. | |Previously, all | |246 subfields were indexed, | |including non-title | |subfields such as $i | |(Display text), $g | |(Miscellaneous | |information), and linking | |subfields, making the title | |index very large and giving | |false results, especially | |when looking for duplicates | |in cataloging. --- Comment #10 from Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com> --- Is the title better, and release notes clear? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37871 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|additional_work_needed, | |release-notes-needed | --- Comment #11 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I like it, thanks :) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37871 Lucas Gass (lukeg) <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lucas@bywatersolutions.com Version(s)|24.11.00 |24.11.00,24.05.06 released in| | Status|Pushed to main |Pushed to stable --- Comment #12 from Lucas Gass (lukeg) <lucas@bywatersolutions.com> --- Backported to 24.05.x for upcoming 24.05.06 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37871 Lucas Gass (lukeg) <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to stable |Pushed to oldstable -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37871 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m Status|Pushed to oldstable |Pushed to oldoldstable Version(s)|24.11.00,24.05.06 |24.11.00,24.05.06,23.11.11 released in| | --- Comment #13 from Fridolin Somers <fridolin.somers@biblibre.com> --- Pushed to 23.11.x for 23.11.11 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37871 Wainui Witika-Park <wainuiwitikapark@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |wainuiwitikapark@catalyst.n | |et.nz Status|Pushed to oldoldstable |Needs documenting --- Comment #14 from Wainui Witika-Park <wainuiwitikapark@catalyst.net.nz> --- Not backporting to 22.11 unless requested -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37871 Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs documenting |RESOLVED Resolution|--- |FIXED --- Comment #15 from Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com> --- Specific indexed fields are not in the manual. No change to the manual needed. -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org