[Bug 43483] New: Authority linker treats the same name heading differently in 6XX vs 1XX/7XX
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43483 Bug ID: 43483 Summary: Authority linker treats the same name heading differently in 6XX vs 1XX/7XX Initiative type: --- Sponsorship --- status: Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: MARC Authority data support Assignee: koha-bugs@lists.koha-community.org Reporter: martin.renvoize@openfifth.co.uk QA Contact: testopia@bugs.koha-community.org Target Milestone: --- When LinkerConsiderThesaurus is enabled, the authority linker treats a name heading differently depending on whether it appears in a name field (100/110/111/700/710/711 etc, where MARC21's 2nd indicator is undefined) or in the equivalent name-used-as-subject field (600/610/611 etc, where the 2nd indicator encodes the subject thesaurus). C4::Heading::MARC21::parse_heading only computes a 'thesaurus' value for 6XX tags: my $thesaurus = $tag =~ m/6../ ? _get_subject_thesaurus($field) : undef; # We can't know the thesaurus for non-subject fields C4::Heading::_search then only adds a thesaurus constraint to the authority match query when $thesaurus is defined: if ( $check_thesaurus && $thesaurus ) { push @marclist, 'thesaurus'; ... } So the same person can end up matched to, or auto-creating, two separate authority records from the same bibliographic record: one for their 100 field (no thesaurus constraint, matches/creates against auth_type PERSO_NAME alone) and a different one for their 600 field (thesaurus constraint applied, even when indicator 2 is blank, which is treated as the placeholder thesaurus 'notdefined'). The two authority records end up with identical headings, so Koha's own duplicate-authority detection later flags them as duplicates when a cataloguer tries to merge or link them manually, even though the automatic linker created them as two separate records in the first place. This is not a new regression. Bug 33277 (2023) deliberately made thesaurus undef for non-6XX tags, to stop 100/700 fields failing to match authorities that had no thesaurus recorded at all. Its commit message notes the trade-off explicitly: "this means that Koha does not support multiple thesaurus records for authorities outside of subjects." Nobody has revisited the resulting asymmetry between a name field and the same auth_type used as a subject field. This was reported by a customer after bug 31925 made the 6XX-vs-thesaurus distinction more reliable (correctly coding 008/11 for FAST/MeSH/etc on newly created authorities), which appears to have made this pre-existing asymmetry more visible: a blank indicator 2 on a 600 is still treated as thesaurus 'notdefined' and still gets the thesaurus filter applied during matching, unlike the 1XX/7XX occurrence of the exact same name. Test plan: 1. Turn on LinkerConsiderThesaurus, AutoLinkBiblios and AutoCreateAuthorities. 2. Catalogue a bibliographic record with the same personal name in both a 100 (or 700) field and a 600 field, e.g.: 100 1_ $a Goddard, Giles 600 1_ $a Goddard, Giles $x Criticism and interpretation (blank/undefined 2nd indicator on the 600, i.e. no thesaurus recorded) 3. Save the record, with no existing matching authority in the catalogue. 4. Note that two separate authority records are created for "Goddard, Giles" -- one from the 100, one from the 600. 5. Try to manually merge or link the two authorities together in the Authorities module. Observe that Koha's duplicate detection considers them duplicates of each other, confirming they should have matched to a single authority in the first place. Related bugs: Bug 31925 -- AutoCreateAuthorities needs to preserve subject heading thesaurus. This report surfaced during discussion of that fix; it is not a regression caused by it. Bug 33277 -- Do not search using thesaurus when linking non 6XX fields. Introduced the asymmetry described above, as an accepted trade-off at the time. Bug 39503 -- Linker should always respect thesaurus with LinkerConsiderThesaurus on. Related but narrower: only addresses collapsing 008/11='z' plus 040$f within 6XX-to-6XX matching. Bug 41714 -- MARC21 authorities thesaurus link hardcoded. Touches the same C4::Heading::MARC21 code (the tag-to-auth_type table), but only makes that mapping configurable via frameworks; it does not touch the thesaurus computation or filtering discussed here. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43483 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=31925, | |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=33277, | |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=39503, | |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=41714 -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43483 cgresser <cgresser@jesuit.org.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |cgresser@jesuit.org.uk --- Comment #1 from cgresser <cgresser@jesuit.org.uk> --- Sorry, I might be reading this wrongly, and might have reported this badly, but the authority heading in 100 and 600, must be different: In our example, the author of the book is writing about himself (a biography), so 100 1_ $a Goddard, Giles,$d1962- 600 1_ $a Goddard, Giles,$d1962- The authority linked from 100 field is only when Rev. Giles Goddard is the writer, the authority linked from 600 field is when Rev. Giles Goddard is being written about. Like I could write a book about him (fear not, I probably won't!), and then would need to use the same authority of 600 1 _ $a Goddard, Giles,$d1962- . However, the 100 in record 1/book 1 and 700 of the record 2/book 2 should link to the same authority. record 2/book 2 might a book where the author has been involved, which could be: 100 1_ $a Randomlynamed, Joe,$d1982 700 1_ $a Goddard, Giles,$d1962- Joe Randomlynamed and Giles Goddard wrote this book together (or were editors). If the authority in our example 1/book 1 for the 100 and the one in our example 2/book 2 for the 700 is the same authority record, then the end user clicking on the link in the OPAC get's books which are either solely written or edited by Giles Goddard (as in 100 and 700). That would make sense to me in Koha. If you want to see an example for this: https://hey.koha.openfifth.net/cgi-bin/koha/opac-search.pl?q=an:141508 books where G.K. Chesteron is the author (100). Books where G.K. is the subject (600) https://hey.koha.openfifth.net/cgi-bin/koha/opac-search.pl?q=an:141508 I might be wrong, but I think that Koha gets confused about these being the same authority. Hope this helps... -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43483 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CONFIRMED -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43483 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |martin.renvoize@openfifth.c |ity.org |o.uk -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43483 Esther Melander <esther.melander@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |esther.melander@bywatersolu | |tions.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43483 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43483 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Comma delimited| |Martin Renvoize list of Sponsors| |<martin.renvoize@gmail.com> Patch complexity|--- |Trivial patch Sponsorship status|--- |Sponsored Status|CONFIRMED |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43483 --- Comment #2 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 205295 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205295&action=edit Bug 43483: Don't treat blank thesaurus indicator as 'notdefined' for matching C4::Heading::MARC21::_get_subject_thesaurus coerced a blank/undefined 2nd indicator on a 6XX field to the literal thesaurus value 'notdefined', while parse_heading leaves the thesaurus undef for the equivalent 1XX/7XX name field. C4::Heading::_search only applies a thesaurus constraint to the authority-matching query when thesaurus is defined, so the same personal name ended up searched/created without any thesaurus constraint via a 100/700 field, but with a hard 'notdefined' constraint via a 600 field with no thesaurus recorded - letting the linker create two separate authority records for the same heading, which Koha's own duplicate-authority detection then flags as duplicates. A blank/undefined 2nd indicator asserts no thesaurus at all, so it should be treated the same as a non-subject field: leave thesaurus undef, and only apply a constraint when a thesaurus is explicitly coded (indicator 0-7). Test plan: 1. Run t/db_dependent/Heading_MARC21.t and t/db_dependent/Heading.t - all tests pass, including new cases asserting no thesaurus is generated for a 6XX field with a blank/undefined 2nd indicator, and that no thesaurus constraint is added to the search query in that case. 2. Turn on LinkerConsiderThesaurus, AutoLinkBiblios and AutoCreateAuthorities. 3. Catalogue a bibliographic record with the same personal name in both a 100 (or 700) field and a 600 field with a blank 2nd indicator, e.g.: 100 1_ $a Goddard, Giles 600 1_ $a Goddard, Giles $x Criticism and interpretation 4. Save the record, with no existing matching authority in the catalogue. 5. Confirm only one authority record is created for "Goddard, Giles", used by both the 100 and the 600. Sponsored-by: Martin Renvoize <martin.renvoize@gmail.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43483 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |43489 Depends on| |31925 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 [Bug 31925] AutoCreateAuthorities needs to preserve subject heading thesaurus to not create duplicate authority records https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43489 [Bug 43489] Thesaurus filtering should be scoped by auth_type, not by MARC21 tag position -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43483 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |phil@chetcolibrary.org -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43483 Gretchen Maxeiner <maxeinergl@gcc.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |maxeinergl@gcc.edu --- Comment #3 from Gretchen Maxeiner <maxeinergl@gcc.edu> --- I am wondering about the selection of blank for 6xx second indicator since this isn't defined for MARC21 (https://www.loc.gov/marc/bibliographic/bd600.html). We would have encoded these names-as-subjects as 2nd indicator 0, (LCSH) which allows for terms from name authority files. Is the blank 2nd indicator a common use? Assuming that use scenario, I did some testing on this today but was unsure on a few spots in the test plan. - step 1 - I'm not sure how to run these tests - step 3 - I think we want to test with the same heading here? "Goddard, Giles" and "Goddard, Giles $x Criticism and interpretation" should result in separate authorities anyway. This confused me on intent. Overall I believe the patch works if the aim is to assure that 1xx/7xx and 6xx (2nd indicator blank) create and map to the same authority records. To see the fix, I created bibliographic records for 4 books as if 2 new authors kept writing biographies about each other, so they alternated between existing in the 100 and 600 fields. (Book 1 - 100 1_ Guy, This. 600 1_ Guy, That. ; Book 2 - 100 1_ Guy, That. 600 1_ Guy, This ; Book 3 - 100 1 Guy, This. 600 1_ Guy, That ; Book 4 - 100 1_ Guy, That. 600 1, Guy, This) Pre-patch this was a mess of 5 authority records and 2 headings left unlinked. After patch the same process resulted in a clean single authority for Guy, This and a clean single authority for Guy, That, with all use instances linked. I subsequently added 700 1_ fields for both Guys on the records and they successfully linked to the two unique authorities. I tested an autobiography scenario, with the same new heading in 100 1_ and 600 1_, and those correctly linked to a single authority both before and after the patch. I have not signed off because of open questions with the intent and test plan, but can if I have a correct understanding! Thanks. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43483 Miranda Nero <mnero@oslri.net> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mnero@oslri.net -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43483 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #205295|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43483 --- Comment #4 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 209025 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=209025&action=edit Bug 43483: Don't treat blank thesaurus indicator as 'notdefined' for matching -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43483 --- Comment #5 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Thanks both for the testing and the questions, and sorry the test plan was unclear. I've replaced the patch (new attachment, no code change) with a rewritten test plan. Author vs. subject authorities: the heading for a person is one entity in a name authority record, wherever it is used. Whether the person is the author (100/700) or the subject (600) is recorded by the field in the bibliographic record, not by having two authority records, and searching on the authority number finds the record in any of those roles. Splitting them in two would stop an "all records for this person" link from working, so the aim of the patch is the opposite: one authority per heading, which the testing here confirms (100 + 700 sharing one authority, and the autobiography case, with the 4-book scenario going from 5 authorities and 2 unlinked headings to 2 clean authorities). Blank 2nd indicator: you are right that it is not defined for 6XX in MARC21 and that 0 is the correct coding for names used as subjects. Blank is what some imported and legacy records carry, and it was being turned into the thesaurus 'notdefined', which then never matched the authority created from the 100. The patch makes blank behave the same as no thesaurus. Records coded 0 are unaffected here (bug 43489 handles that case). Test plan: step 1 now gives the exact `prove` command (and how to see the tests fail without the fix), and the scenario uses identical headings in the 100 and 600 (the autobiography case), followed by the 700 case. I also checked this end to end against Elasticsearch with LinkerConsiderThesaurus on: a name with a blank 2nd indicator links to the single existing name authority. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43483 --- Comment #6 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 209025 (full commit message, for the test plan) Bug 43483: Don't treat blank thesaurus indicator as 'notdefined' for matching C4::Heading::MARC21::_get_subject_thesaurus coerced a blank/undefined 2nd indicator on a 6XX field to the literal thesaurus value 'notdefined', while parse_heading leaves the thesaurus undef for the equivalent 1XX/7XX name field. C4::Heading::_search only applies a thesaurus constraint to the authority-matching query when thesaurus is defined, so the same personal name ended up searched/created without any thesaurus constraint via a 100/700 field, but with a hard 'notdefined' constraint via a 600 field with no thesaurus recorded - letting the linker create two separate authority records for the same heading, which Koha's own duplicate-authority detection then flags as duplicates. A blank/undefined 2nd indicator asserts no thesaurus at all, so it should be treated the same as a non-subject field: leave thesaurus undef, and only apply a constraint when a thesaurus is explicitly coded (indicator 0-7). Note that a 6XX 2nd indicator is not defined as blank in MARC21 for name headings, but blank is what many imported and legacy records carry. Records using 2nd indicator 0 are unaffected by this patch. To test: 1. Run the unit tests inside KTD: ktd --shell --run 'prove t/db_dependent/Heading_MARC21.t t/db_dependent/Heading.t' They pass. To see them fail, revert only the change to C4/Heading/MARC21.pm and run them again: the new blank-indicator cases fail. 2. Enable the LinkerConsiderThesaurus, AutoLinkBiblios and AutoCreateAuthorities system preferences. 3. Make sure there is no authority for "Goddard, Giles" yet. 4. Catalogue a new bibliographic record with the same name as author and as subject (an autobiography), leaving the 6XX 2nd indicator blank: 100 1_ $a Goddard, Giles $d 1962- 600 1_ $a Goddard, Giles $d 1962- 5. Save the record, then search Authorities for "Goddard". - Without this patch: two authority records are created for the same heading. - With this patch: a single authority record, linked from both the 100 and the 600. 6. Catalogue a second record with a different author and 700 1_ $a Goddard, Giles $d 1962- Save it and confirm it links to that same authority record, and that no further authority was created. 7. Sign off Sponsored-by: Martin Renvoize <martin.renvoize@gmail.com> -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org