[Bug 43489] New: Thesaurus filtering should be scoped by auth_type, not by MARC21 tag position
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43489 Bug ID: 43489 Summary: Thesaurus filtering should be scoped by auth_type, not by MARC21 tag position Initiative type: --- Sponsorship --- status: Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: MARC Authority data support Assignee: koha-bugs@lists.koha-community.org Reporter: martin.renvoize@openfifth.co.uk QA Contact: testopia@bugs.koha-community.org Depends on: 43483 Target Milestone: --- Bug 43483 fixed the most common case of this problem (a blank/undefined 2nd indicator on a 6XX field being coerced to the literal thesaurus value 'notdefined'). This bug tracks the more thorough underlying fix. C4::Heading::MARC21::parse_heading decides whether to compute a thesaurus value purely from the MARC tag position: my $thesaurus = $tag =~ m/6../ ? _get_subject_thesaurus($field) : undef; That means any 6XX occurrence of a name heading (100/110/111/130 used as 600/610/611/630) can still get a thesaurus constraint applied during authority matching whenever indicator 2 is explicitly coded (0-7), even after bug 43483's fix for the blank-indicator case. For example: 100 1_ $a Goddard, Giles 600 10 $a Goddard, Giles $x Criticism and interpretation (indicator 2 = 0, i.e. lcsh explicitly coded) will still search/create against a 'lcsh' thesaurus constraint for the 600 occurrence, while the 100 occurrence has no constraint at all. If the authority created via the 100 field was not recorded with an explicit lcsh thesaurus, the two can again end up as separate authority records, same symptom as bug 43483, just via a narrower path. The real distinction is not "is this tag a 6XX" but "does this heading's auth_type represent a controlled vocabulary term, or a named real-world entity". MARC's indicator-2 thesaurus code exists to disambiguate genuinely separate subject vocabularies (LCSH vs MeSH vs Sears etc) that may define overlapping terms. It has no equivalent meaning for personal names, corporate names, meeting names or uniform titles: NACO name authority records are reused identically regardless of whether the name functions as a main entry (1XX), added entry (7XX/8XX) or subject added entry (6XX) in a given bibliographic record. There is exactly one authority record per real-world entity, not one per (entity, thesaurus) pair. Suggested solution: Replace the tag-position check in parse_heading with an auth_type check. Introduce a small table of auth_types for which a thesaurus is a meaningful concept - genuine subject/vocabulary terms: TOPIC_TERM, GEOGR_NAME, CHRON_TERM, GENRE/FORM, MED_PERFRM and only call _get_subject_thesaurus for a 6XX field whose auth_type is in that set. Name-entity auth_types - PERSO_NAME, CORPO_NAME, MEETI_NAME, UNIF_TITLE, NAME_EVENT - would then never get a thesaurus constraint, in any tag position, even when indicator 2 is explicitly coded. This is more thorough and correct than the blank-indicator fix in bug 43483 because: 1. It closes the remaining gap: an explicitly coded thesaurus indicator on a name-type 6XX field no longer fragments matching either, not just the blank/undefined case. 2. It matches real cataloguing practice (NACO/LCNAF): the thesaurus concept is a property of subject vocabularies, not of names. Scoping by auth_type encodes that directly instead of relying on tag position as a proxy for it. 3. It is symmetric by construction for every current and future name auth_type, rather than requiring each edge case (blank indicator, explicit indicator, UNIMARC vs MARC21) to be patched individually as it is discovered, which is how bug 33277 and bug 43483 each arose. 4. Genuine subject/vocabulary auth_types (TOPIC_TERM, GEOGR_NAME, CHRON_TERM, GENRE/FORM, MED_PERFRM) keep their existing, correct thesaurus-scoped behaviour, so this does not reopen bug 33277 or affect bug 39503's 6XX-to-6XX collapsing logic. C4::Heading::UNIMARC::parse_heading already always returns thesaurus undef, so UNIMARC is unaffected by either bug and needs no change. Test plan: 1. Turn on LinkerConsiderThesaurus, AutoLinkBiblios and AutoCreateAuthorities. 2. Catalogue a bibliographic record with the same personal name in a 100 field and a 600 field where indicator 2 is explicitly coded (e.g. 0 for lcsh), with no existing matching authority. 3. Confirm only one authority record is created and used by both fields, the same as the blank-indicator case fixed by bug 43483. 4. Confirm existing subject heading matching for TOPIC_TERM, GEOGR_NAME, CHRON_TERM, GENRE/FORM and MED_PERFRM auth_types is unchanged (t/db_dependent/Heading.t "_search tests"). Related bugs: Bug 43483 -- Authority linker treats the same name heading differently in 6XX vs 1XX/7XX. This bug depends on it and generalises its fix from "blank indicator" to "any indicator, for name-type auth_types". Bug 33277 -- Do not search using thesaurus when linking non 6XX fields. The auth_type-based check keeps this behaviour for name-type auth_types, so this is not reopened. Bug 39503 -- Linker should always respect thesaurus with LinkerConsiderThesaurus on. Addresses 6XX-to-6XX matching for subject/vocabulary auth_types, unaffected by this proposal. Bug 41714 -- MARC21 authorities thesaurus link hardcoded. Makes the tag-to-auth_type table configurable via frameworks; the auth_type-based thesaurus check proposed here would need to read from that configurable mapping rather than the hardcoded one, if 41714 lands first. Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43483 [Bug 43483] Authority linker treats the same name heading differently in 6XX vs 1XX/7XX -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43489 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=33277, | |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=39503, | |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=41714 -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43489 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |martin.renvoize@openfifth.c |ity.org |o.uk -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43489 --- Comment #1 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 205296 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205296&action=edit Bug 43489: Scope thesaurus filtering by auth_type, not by MARC21 tag position C4::Heading::MARC21::parse_heading decided whether to compute a thesaurus constraint purely from tag position ($tag =~ m/6../). Bug 43483 fixed the most common trigger of this (a blank/undefined 2nd indicator being coerced to the literal 'notdefined' thesaurus), but any 6XX occurrence of a name heading with an explicitly coded indicator 2 (0-7) could still get a thesaurus constraint applied during authority matching, while the equivalent 1XX/7XX occurrence of the same name never does - so the same personal, corporate or meeting name could still end up matched/created against two separate authority records depending on which field it appeared in and how indicator 2 was coded. The thesaurus concept only has meaning for genuine controlled subject vocabularies (LCSH vs MeSH vs Sears etc) that may define overlapping terms - it has no equivalent meaning for named real-world entities. NACO name authority records are reused identically regardless of whether the name functions as a main entry, added entry, or subject added entry. This introduces a table of auth_types for which a thesaurus is a meaningful concept (TOPIC_TERM, GEOGR_NAME, CHRON_TERM, GENRE/FORM, MED_PERFRM) and only computes/applies a thesaurus constraint for a 6XX field whose auth_type is in that set. Name-entity auth_types (PERSO_NAME, CORPO_NAME, MEETI_NAME, UNIF_TITLE, NAME_EVENT) never get a thesaurus constraint, in any tag position, regardless of how indicator 2 is coded. Test plan: 1. Run t/db_dependent/Heading_MARC21.t and t/db_dependent/Heading.t - all tests pass, including new cases confirming a PERSO_NAME/ CORPO_NAME 6XX field with an explicitly coded thesaurus indicator still gets no thesaurus constraint, while a TOPIC_TERM 6XX field's existing thesaurus-scoped behaviour is unchanged. 2. Turn on LinkerConsiderThesaurus, AutoLinkBiblios and AutoCreateAuthorities. 3. Catalogue a bibliographic record with the same personal name in a 100 field and a 600 field with indicator 2 explicitly coded (e.g. 0 for lcsh), with no existing matching authority. 4. Confirm only one authority record is created, used by both fields. 5. Run t/db_dependent/Linker_Default.t, t/db_dependent/Linker_FirstMatch.t, t/db_dependent/AuthoritiesMarc.t and t/db_dependent/Koha/Authority.t - all pass, no regressions. Depends on Bug 43483. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43489 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43489 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |cgresser@jesuit.org.uk -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43489 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |phil@chetcolibrary.org -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43489 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43489 Phil Ringnalda <phil@chetcolibrary.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA --- Comment #2 from Phil Ringnalda <phil@chetcolibrary.org> --- It's not as simple as the auth_type, since 600 1 7 $aGoddard, Giles$vNovela$2bidex is both unambiguously thesaurus-scoped and not something that would have been autocreated from a 100 tag. Possibly it's as simple as "is a 600 and contains no subfields which are in $bib_heading_fields->600->subfields but not in $bib_heading_fields->100->subfields"? Not entirely sure, since that's not exactly what those lists were created for and I haven't examined every single subfield. And should we be limiting this to only when AutoCreateAuthorities is on, since it's solving a problem that doesn't exist when it's off? Consider a library with a population that is half Japanese-speaking and half English-speaking, which catalogs all its works in both LCSH and NDL and chooses which to display based on the language each user has chosen. They will not thank you for linking both of their Shakespeare, William, 1564-1616 600s to one authority record. Or for treating them as duplicates by doing a non-thesaurus-scoped search for them. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43489 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #205296|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43489 --- Comment #3 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 209024 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=209024&action=edit Bug 43489: Scope thesaurus filtering by auth_type, not by MARC21 tag position -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43489 --- Comment #4 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Thanks Phil, both points were right. I've reworked the patch (new attachment, also rebased on the updated bug 43483). Your $2 bidex example: the blanket rule dropped the thesaurus for every name heading, which would have merged names that explicitly name another vocabulary. The rule is now narrower. For a 6XX field whose authority type is not a thesaurus-scoped one (names, uniform titles), only a thesaurus that merely says "Library of Congress regime" is discarded: lcsh (2nd indicator 0), lcac (1) and "not specified" (4), because name authorities live in that regime and are the same entity wherever the name is used. A name with 2nd indicator 2, 3, 5, 6 or 7 plus $2 keeps its thesaurus, so 600 17 $a Goddard, Giles $v Novela $2 bidex is still scoped to bidex. Topical, geographic, chronological, genre/form and performance-medium headings are unchanged. (Names carrying $v/$x subdivisions already produce a different heading string from the 100, so they were never at risk of merging with it.) Your bilingual LCSH + NDL (or MeSH, etc.) library: those 6XX fields have 2nd indicator 7 with $2, or 2/3/5/6, so they keep their thesaurus both when matching and when creating, and they are not merged with each other or with the LC record. Only the LC-regime names are matched without a thesaurus, which is what a 100/700 already does. On limiting this to AutoCreateAuthorities: the thesaurus constraint is also applied to the search the linker runs against existing authorities when LinkerConsiderThesaurus is on, so a 600 with 2nd indicator 0 failed to link to a name authority that a 100 links to regardless of AutoCreateAuthorities. So I have not gated it on that preference, but I'm happy to if you think that is better. Tests: new cases cover 600/610 with 2nd indicator 0, 1 and 4 (no thesaurus), 2, 5 and 6 (thesaurus kept), a $2 bidex name (kept) and the topical 650 cases, plus the _search query for the bidex name. The cases for 0, 1 and 4 fail without the fix. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43489 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43489 --- Comment #5 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 209024 (full commit message, for the test plan) Bug 43489: Scope thesaurus filtering by auth_type, not by MARC21 tag position C4::Heading::MARC21::parse_heading decided whether to compute a thesaurus constraint purely from tag position ($tag =~ m/6../). Bug 43483 fixed the most common trigger of this (a blank/undefined 2nd indicator being coerced to the literal 'notdefined' thesaurus), but a 6XX name heading with a coded 2nd indicator of 0, 1 or 4 still got a thesaurus constraint ('lcsh', 'lcac' or 'notspecified') applied during authority matching, while the equivalent 1XX/7XX occurrence of the same name never does. The same personal, corporate or meeting name could therefore still be matched or created against two separate authority records depending on which field it appeared in. Those three values only say that the heading comes from the Library of Congress regime, which is where name authorities (NACO) live. They say nothing about a name: it is the same entity wherever it is used, as author, added entry or subject. This introduces a table of auth_types for which a thesaurus is a meaningful concept (TOPIC_TERM, GEOGR_NAME, CHRON_TERM, GENRE/FORM, MED_PERFRM), and a table of the thesauri that only describe the LC regime (lcsh, lcac, notspecified). For a 6XX field whose auth_type is not thesaurus-scoped, a thesaurus drawn from that second table is discarded. A name that explicitly names another vocabulary (2nd indicator 2, 3, 5, 6 or 7 with $2) keeps its thesaurus, so a heading taken from, for example, MeSH or a local source is still not merged with the LC one. Test plan: 1. Run the unit tests inside KTD: ktd --shell --run 'prove t/db_dependent/Heading_MARC21.t t/db_dependent/Heading.t' They pass. To see them fail, revert only the change to C4/Heading/MARC21.pm and run them again: the cases for 600/610 with 2nd indicator 0, 1 and 4 fail. 2. Enable the LinkerConsiderThesaurus, AutoLinkBiblios and AutoCreateAuthorities system preferences. 3. Make sure there is no authority for "Goddard, Giles" yet. 4. Catalogue a new bibliographic record with the same name in a 100 and in a 600 with the 2nd indicator coded 0 (LCSH): 100 1_ $a Goddard, Giles $d 1962- 600 10 $a Goddard, Giles $d 1962- 5. Save it and search Authorities for "Goddard". - Without this patch: two authority records. - With this patch: one authority record, linked from both fields. 6. Repeat with the 600 as 600 14 $a Goddard, Giles $d 1962- and confirm it is linked to that same authority record too. 7. Run the other linker tests and confirm there are no regressions: ktd --shell --run 'prove t/db_dependent/Linker_Default.t t/db_dependent/Linker_FirstMatch.t t/db_dependent/AuthoritiesMarc.t t/db_dependent/Koha/Authority.t' 8. Sign off Depends on Bug 43483. -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org