[Bug 31925] New: AutoCreateAuthorities needs to preserve subject heading thesaurus to not create duplicate authority records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 Bug ID: 31925 Summary: AutoCreateAuthorities needs to preserve subject heading thesaurus to not create duplicate authority records Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: major Priority: P5 - low Component: MARC Authority data support Assignee: koha-bugs@lists.koha-community.org Reporter: phil@chetcolibrary.org QA Contact: testopia@bugs.koha-community.org CC: nick@bywatersolutions.com Depends on: 30280 Bug 30280 taught the linker that "650 _ 0 $aFeminism" and "650 _ 7 $aFeminism$2sao" are not the same heading, so a bib record containing the latter should not be linked to the authority record which matches the former. However, AutoCreateAuthorities just creates LCSH authority records from anything, no matter what the values of indicator 2 and subfield 2 are. As a result, if you have an LCSH authority record for a term, and you import/create a non-LCSH term in a bib record and save it, AutoCreateAuthorities will create a new duplicate LCSH authority record and link to that. Every single time, a new duplicate. Steps to reproduce: 1. Use ElasticSearch, since the patch for bug 30280 only expected to work there, and in fact broke linking on Zebra, bug 31924. 2. Enable the preferences AutoCreateAuthorities, AutoLinkBiblios, CatalogModuleRelink 3. Edit any bib record, add a 650 tag, set the second indicator to 7, type Feminism in subfield a and type sao in subfield 2. 4. Save the record, edit it again, note the authority record id in subfield 9 of the Feminism heading. 5. Edit any other bib record, add a 650 tag, set the second indicator to 7, type Feminism in subfield a and type sao in subfield 2. 6. Save the record, edit it again, note the authority record id in subfield 9 of the Feminism heading. The two authority record ids should be the same, since the headings are identical, but instead one will be linked to, say, 1708 and the next to 1709, an identical duplicate record. Rather than setting position 11 of field 008 in the authority record to "a" for LCSH, AutoCreateAuthorities should have set it to "z" for Other and put sao in subfield f of field 040 (and for non-7, non-0 values of indicator 2, it should be setting 008/11 to the correct letter for that indicator value). Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30280 [Bug 30280] Support authority records with common subject headings from different thesaurus -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 Frank Hansen <frank.hansen@ub.lu.se> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |frank.hansen@ub.lu.se -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 --- Comment #1 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Could we get some attention to this one? -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 --- Comment #2 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Just some thought: I think Phil's suggestion for making Koha create correct records is the right one. But also: what do we do with the existing 'bad' records. Could it be an option to make the new stricter behavior dependent on a system preference? -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 Bug 31925 depends on bug 30280, which changed state. Bug 30280 Summary: Support authority records with common subject headings from different thesaurus https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30280 What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to master |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 Agnes Rivers-Moore <arm@hanover.ca> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |arm@hanover.ca --- Comment #3 from Agnes Rivers-Moore <arm@hanover.ca> --- Definitely interested as a librarian/user - we recently noticed that Authorities are duplicating out of control! Sorry I can't help with coding. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=33557 -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 Agnes Rivers-Moore <arm@hanover.ca> changed: What |Removed |Added ---------------------------------------------------------------------------- CC|arm@hanover.ca | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 Phil Ringnalda <phil@chetcolibrary.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |cgresser@jesuit.org.uk --- Comment #4 from Phil Ringnalda <phil@chetcolibrary.org> --- *** Bug 42694 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 --- Comment #5 from cgresser <cgresser@jesuit.org.uk> --- Could this please be tackled soon? It is really adding a lot (!) of inefficiency to Koha, and many cataloguers in the English-speaking (Marc21) world will love you for fixing this. I almost constantly have to deal with false-positive duplicates, having to merge or delete authorities. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |martin.renvoize@openfifth.c | |o.uk --- Comment #6 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- We're scoping a fix for this at Open Fifth. Our read: since bug 30280 made the linker thesaurus-aware (008/11 + 040$f) but AutoCreateAuthorities was never updated to match, every auto-created record for a non-LCSH heading gets miscoded as LCSH, so it can never be matched next time — producing an unbounded stream of duplicates (see bug 42694 for a real-world case). We're planning to align AutoCreateAuthorities with the same thesaurus-derivation logic the linker already uses for matching. Will follow up with a fuller proposal, including whether this should sit behind a preference given the precedent set by bug 33557. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |martin.renvoize@openfifth.c |ity.org |o.uk -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 --- Comment #8 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 201861 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201861&action=edit Bug 31925: Make QueryBuilder reuse the shared thesaurus/008 lookup Koha::SearchEngine::Elasticsearch::QueryBuilder kept its own copy of the thesaurus-name to 008/11-code table used when building "thesaurus" search queries against authority records. The previous commit moved that same table to Koha::Authority as a shared lookup. This commit deletes QueryBuilder's duplicate and points $thesaurus_to_value at $Koha::Authority::MARC21_THESAURUS_TO_CONTROL_FIELD_008_11 instead, so there is only one place to update if the code list ever changes. Pure refactor: $thesaurus_to_value keeps its existing name and contents, and build_authorities_query_compat() is unchanged, so query behaviour is identical before and after. Sponsored-by: Heythrop Library <https://heythroplibrary.co.uk/> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Comma delimited| |Heythrop Library list of Sponsors| |<https://heythroplibrary.co | |.uk/> Patch complexity|--- |Small patch Status|ASSIGNED |Needs Signoff Sponsorship status|--- |Sponsored -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 --- Comment #9 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 201862 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201862&action=edit Bug 31925: AutoCreateAuthorities must preserve subject heading thesaurus When AutoCreateAuthorities creates a new authority for a bib subject heading that didn't match an existing one, it never set 008 or 040 on the new MARC::Record, so C4::AuthoritiesMarc::AddAuthority()'s own defaulting always ran and hardcoded 008/11 to 'a' (LCSH), regardless of the source heading's real thesaurus. A FAST, MeSH, or other non-LCSH heading (MARC indicator 2 = 7 with subfield $2, or indicator 2 = 1-6) therefore always got mis-coded as LCSH on creation. Because the authority was permanently mis-coded, it could never be found again by thesaurus-aware matching (LinkerConsiderThesaurus), so every later import of the same non-LCSH heading created yet another duplicate authority instead of linking to the one already created (see bug 42694 for a real-world report of this). This calls Koha::Authority->default_marc21_008($thesaurus) to build the new authority's 008 - correctly coding position 11 for the source heading's actual thesaurus - and, for headings identified only by a raw $2 code, adds a 040$f before calling AddAuthority(), whose existing "only default if the field isn't already there" guards then leave our correctly-coded fields alone. No change to AddAuthority() itself is needed for this. This is MARC21-only (matches the existing marcflavour branch already in this code path) and ships as an unconditional bug fix, with no new system preference: it only changes how a *new* authority is coded going forward and does not touch or merge any existing authority records. Test plan (librarian-facing): 1. Turn on the AutoLinkBiblios and AutoCreateAuthorities system preferences (Administration -> System preferences -> Cataloging). 2. Catalog a new bibliographic record, or edit an existing one, and add a 650 field coded for a non-LCSH thesaurus, e.g.: 650 _7 $a Miracles $2 fast (2nd indicator = 7, subfield $2 = fast; a FAST heading with no existing matching authority in your catalogue). 3. Save the record. Koha auto-creates a new "Miracles" authority record, since none existed yet to link to. 4. In the Authorities module, open the newly created "Miracles" record for editing and switch to MARC view (Edit -> Edit record, or the "..." menu -> Edit as new). 5. Click into the 008 field to open its editor plugin. Before this fix: the "11- Subject heading system/thesaurus" dropdown is set to "a- Library of Congress Subject Headings", and field 040 has no $f subfield. After this fix: that dropdown is now set to "z- Other", and field 040 has a $f subfield containing "fast" — correctly recording the real source vocabulary instead of silently mis-labelling it as LCSH. 6. As a regression check, repeat steps 2-5 with a plain LCSH heading instead, e.g.: 650 __0 $a Feminism (2nd indicator = 0, no $2). Confirm the resulting authority's 008 still shows "a- Library of Congress Subject Headings" as before — this fix must not change behaviour for ordinary LCSH headings. 7. Optional, Elasticsearch installations only: with LinkerConsiderThesaurus also turned on, re-import or re-save a bib with the same FAST heading from step 2 a second time. It should now link to the authority created in step 3 instead of creating a second duplicate — this is the actual duplicate-authority problem reported in bug 42694. Sponsored-by: Heythrop Library <https://heythroplibrary.co.uk/> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 --- Comment #7 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 201860 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201860&action=edit Bug 31925: Add thesaurus/008 lookup and default MARC21 008 builder to Koha::Authority AutoCreateAuthorities builds a brand-new authority record whenever a bib subject heading (6XX) has no match, but always lets the record fall back to C4::AuthoritiesMarc::AddAuthority()'s default 008/040, which hardcodes 008 position 11 ("Subject heading system/thesaurus") to 'a' (LCSH) no matter what thesaurus the source heading actually uses. This is the first step towards fixing that. Adds two things to Koha::Authority: - $MARC21_THESAURUS_TO_CONTROL_FIELD_008_11: a single-source-of-truth lookup mapping a thesaurus name (as already derived by C4::Heading from MARC indicator 2 / subfield $2, e.g. 'lcsh', 'mesh', or a raw $2 code such as 'fast') to its MARC21 authority 008/11 code, per the LOC ad008 code list. This table previously only existed once, for a different purpose, in Koha::SearchEngine::Elasticsearch::QueryBuilder (used for building thesaurus search queries) - it's promoted here so both that and the authority-creation fix that follows can read from one place instead of two copies that can silently drift apart. - default_marc21_008($thesaurus): builds the default 34-character body of a new MARC21 authority's 008 field, honouring the MARCAuthorityControlField008 system preference. When $thesaurus is given, position 11 is set from the table above instead of whatever the default/syspref value carries, falling back to 'z' (Other) for a thesaurus with no dedicated code. AddAuthority() is refactored to build its own (thesaurus-less, LCSH default) 008 via this new method too, replacing its previous inline copy of the same defaulting logic - pure refactor, no behaviour change. Nothing thesaurus-aware calls it yet. Sponsored-by: Heythrop Library <https://heythroplibrary.co.uk/> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 Phil Ringnalda <phil@chetcolibrary.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #201860|0 |1 is obsolete| | Attachment #201861|0 |1 is obsolete| | Attachment #201862|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 --- Comment #10 from Phil Ringnalda <phil@chetcolibrary.org> --- Created attachment 201871 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201871&action=edit Bug 31925: Add thesaurus/008 lookup and default MARC21 008 builder to Koha::Authority AutoCreateAuthorities builds a brand-new authority record whenever a bib subject heading (6XX) has no match, but always lets the record fall back to C4::AuthoritiesMarc::AddAuthority()'s default 008/040, which hardcodes 008 position 11 ("Subject heading system/thesaurus") to 'a' (LCSH) no matter what thesaurus the source heading actually uses. This is the first step towards fixing that. Adds two things to Koha::Authority: - $MARC21_THESAURUS_TO_CONTROL_FIELD_008_11: a single-source-of-truth lookup mapping a thesaurus name (as already derived by C4::Heading from MARC indicator 2 / subfield $2, e.g. 'lcsh', 'mesh', or a raw $2 code such as 'fast') to its MARC21 authority 008/11 code, per the LOC ad008 code list. This table previously only existed once, for a different purpose, in Koha::SearchEngine::Elasticsearch::QueryBuilder (used for building thesaurus search queries) - it's promoted here so both that and the authority-creation fix that follows can read from one place instead of two copies that can silently drift apart. - default_marc21_008($thesaurus): builds the default 34-character body of a new MARC21 authority's 008 field, honouring the MARCAuthorityControlField008 system preference. When $thesaurus is given, position 11 is set from the table above instead of whatever the default/syspref value carries, falling back to 'z' (Other) for a thesaurus with no dedicated code. AddAuthority() is refactored to build its own (thesaurus-less, LCSH default) 008 via this new method too, replacing its previous inline copy of the same defaulting logic - pure refactor, no behaviour change. Nothing thesaurus-aware calls it yet. Sponsored-by: Heythrop Library <https://heythroplibrary.co.uk/> Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 --- Comment #12 from Phil Ringnalda <phil@chetcolibrary.org> --- Created attachment 201873 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201873&action=edit Bug 31925: AutoCreateAuthorities must preserve subject heading thesaurus When AutoCreateAuthorities creates a new authority for a bib subject heading that didn't match an existing one, it never set 008 or 040 on the new MARC::Record, so C4::AuthoritiesMarc::AddAuthority()'s own defaulting always ran and hardcoded 008/11 to 'a' (LCSH), regardless of the source heading's real thesaurus. A FAST, MeSH, or other non-LCSH heading (MARC indicator 2 = 7 with subfield $2, or indicator 2 = 1-6) therefore always got mis-coded as LCSH on creation. Because the authority was permanently mis-coded, it could never be found again by thesaurus-aware matching (LinkerConsiderThesaurus), so every later import of the same non-LCSH heading created yet another duplicate authority instead of linking to the one already created (see bug 42694 for a real-world report of this). This calls Koha::Authority->default_marc21_008($thesaurus) to build the new authority's 008 - correctly coding position 11 for the source heading's actual thesaurus - and, for headings identified only by a raw $2 code, adds a 040$f before calling AddAuthority(), whose existing "only default if the field isn't already there" guards then leave our correctly-coded fields alone. No change to AddAuthority() itself is needed for this. This is MARC21-only (matches the existing marcflavour branch already in this code path) and ships as an unconditional bug fix, with no new system preference: it only changes how a *new* authority is coded going forward and does not touch or merge any existing authority records. Test plan (librarian-facing): 1. Turn on the AutoLinkBiblios and AutoCreateAuthorities system preferences (Administration -> System preferences -> Cataloging). 2. Catalog a new bibliographic record, or edit an existing one, and add a 650 field coded for a non-LCSH thesaurus, e.g.: 650 _7 $a Miracles $2 fast (2nd indicator = 7, subfield $2 = fast; a FAST heading with no existing matching authority in your catalogue). 3. Save the record. Koha auto-creates a new "Miracles" authority record, since none existed yet to link to. 4. In the Authorities module, open the newly created "Miracles" record for editing and switch to MARC view (Edit -> Edit record, or the "..." menu -> Edit as new). 5. Click into the 008 field to open its editor plugin. Before this fix: the "11- Subject heading system/thesaurus" dropdown is set to "a- Library of Congress Subject Headings", and field 040 has no $f subfield. After this fix: that dropdown is now set to "z- Other", and field 040 has a $f subfield containing "fast" — correctly recording the real source vocabulary instead of silently mis-labelling it as LCSH. 6. As a regression check, repeat steps 2-5 with a plain LCSH heading instead, e.g.: 650 __0 $a Feminism (2nd indicator = 0, no $2). Confirm the resulting authority's 008 still shows "a- Library of Congress Subject Headings" as before — this fix must not change behaviour for ordinary LCSH headings. 7. Optional, Elasticsearch installations only: with LinkerConsiderThesaurus also turned on, re-import or re-save a bib with the same FAST heading from step 2 a second time. It should now link to the authority created in step 3 instead of creating a second duplicate — this is the actual duplicate-authority problem reported in bug 42694. Sponsored-by: Heythrop Library <https://heythroplibrary.co.uk/> Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 --- Comment #11 from Phil Ringnalda <phil@chetcolibrary.org> --- Created attachment 201872 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201872&action=edit Bug 31925: Make QueryBuilder reuse the shared thesaurus/008 lookup Koha::SearchEngine::Elasticsearch::QueryBuilder kept its own copy of the thesaurus-name to 008/11-code table used when building "thesaurus" search queries against authority records. The previous commit moved that same table to Koha::Authority as a shared lookup. This commit deletes QueryBuilder's duplicate and points $thesaurus_to_value at $Koha::Authority::MARC21_THESAURUS_TO_CONTROL_FIELD_008_11 instead, so there is only one place to update if the code list ever changes. Pure refactor: $thesaurus_to_value keeps its existing name and contents, and build_authorities_query_compat() is unchanged, so query behaviour is identical before and after. Sponsored-by: Heythrop Library <https://heythroplibrary.co.uk/> Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 Phil Ringnalda <phil@chetcolibrary.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lisette@bywatersolutions.co | |m QA Contact|testopia@bugs.koha-communit |lisette@bywatersolutions.co |y.org |m -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|lisette@bywatersolutions.co |dcook@prosentient.com.au |m | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 --- Comment #13 from cgresser <cgresser@jesuit.org.uk> --- I'm not sure whether it's custom to do this, but we have been testing the patch for this bug since yesterday, and it works like a charm: 650 _ 0 ‡aChildbirth‡xMoral and ethical aspects.‡9217329 650 _ 7 ‡aChildbirth‡xMoral and ethical aspects‡2fast‡9217330 Both were new subject headings / access points for us, and they were promptly created by Koha for our local instance/local authorities database. The following is an example of an existing standard LCC heading, and then Koha picked up that the fast one needs to be added to our local authorities: 650 _ 0 ‡aPregnancy‡xMoral and ethical aspects.‡9135277 650 _ 7 ‡aPregnancy‡xMoral and ethical aspects‡2fast‡9217331 It's like Christmas has come early! Now to fixing our local heritage authorities data, which is not Koha's fault. -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org