https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 --- Comment #1 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Could we get some attention to this one? -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 Bug 31925 depends on bug 30280, which changed state. Bug 30280 Summary: Support authority records with common subject headings from different thesaurus https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30280 What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to master |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=33557 -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 Phil Ringnalda <phil@chetcolibrary.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |cgresser@jesuit.org.uk --- Comment #4 from Phil Ringnalda <phil@chetcolibrary.org> --- *** Bug 42694 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |martin.renvoize@openfifth.c | |o.uk --- Comment #6 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- We're scoping a fix for this at Open Fifth. Our read: since bug 30280 made the linker thesaurus-aware (008/11 + 040$f) but AutoCreateAuthorities was never updated to match, every auto-created record for a non-LCSH heading gets miscoded as LCSH, so it can never be matched next time — producing an unbounded stream of duplicates (see bug 42694 for a real-world case). We're planning to align AutoCreateAuthorities with the same thesaurus-derivation logic the linker already uses for matching. Will follow up with a fuller proposal, including whether this should sit behind a preference given the precedent set by bug 33557. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 --- Comment #7 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 201860 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201860&action=edit Bug 31925: Add thesaurus/008 lookup and default MARC21 008 builder to Koha::Authority AutoCreateAuthorities builds a brand-new authority record whenever a bib subject heading (6XX) has no match, but always lets the record fall back to C4::AuthoritiesMarc::AddAuthority()'s default 008/040, which hardcodes 008 position 11 ("Subject heading system/thesaurus") to 'a' (LCSH) no matter what thesaurus the source heading actually uses. This is the first step towards fixing that. Adds two things to Koha::Authority: - $MARC21_THESAURUS_TO_CONTROL_FIELD_008_11: a single-source-of-truth lookup mapping a thesaurus name (as already derived by C4::Heading from MARC indicator 2 / subfield $2, e.g. 'lcsh', 'mesh', or a raw $2 code such as 'fast') to its MARC21 authority 008/11 code, per the LOC ad008 code list. This table previously only existed once, for a different purpose, in Koha::SearchEngine::Elasticsearch::QueryBuilder (used for building thesaurus search queries) - it's promoted here so both that and the authority-creation fix that follows can read from one place instead of two copies that can silently drift apart. - default_marc21_008($thesaurus): builds the default 34-character body of a new MARC21 authority's 008 field, honouring the MARCAuthorityControlField008 system preference. When $thesaurus is given, position 11 is set from the table above instead of whatever the default/syspref value carries, falling back to 'z' (Other) for a thesaurus with no dedicated code. AddAuthority() is refactored to build its own (thesaurus-less, LCSH default) 008 via this new method too, replacing its previous inline copy of the same defaulting logic - pure refactor, no behaviour change. Nothing thesaurus-aware calls it yet. Sponsored-by: Heythrop Library <https://heythroplibrary.co.uk/> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Comma delimited| |Heythrop Library list of Sponsors| |<https://heythroplibrary.co | |.uk/> Patch complexity|--- |Small patch Status|ASSIGNED |Needs Signoff Sponsorship status|--- |Sponsored -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 --- Comment #12 from Phil Ringnalda <phil@chetcolibrary.org> --- Created attachment 201873 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201873&action=edit Bug 31925: AutoCreateAuthorities must preserve subject heading thesaurus When AutoCreateAuthorities creates a new authority for a bib subject heading that didn't match an existing one, it never set 008 or 040 on the new MARC::Record, so C4::AuthoritiesMarc::AddAuthority()'s own defaulting always ran and hardcoded 008/11 to 'a' (LCSH), regardless of the source heading's real thesaurus. A FAST, MeSH, or other non-LCSH heading (MARC indicator 2 = 7 with subfield $2, or indicator 2 = 1-6) therefore always got mis-coded as LCSH on creation. Because the authority was permanently mis-coded, it could never be found again by thesaurus-aware matching (LinkerConsiderThesaurus), so every later import of the same non-LCSH heading created yet another duplicate authority instead of linking to the one already created (see bug 42694 for a real-world report of this). This calls Koha::Authority->default_marc21_008($thesaurus) to build the new authority's 008 - correctly coding position 11 for the source heading's actual thesaurus - and, for headings identified only by a raw $2 code, adds a 040$f before calling AddAuthority(), whose existing "only default if the field isn't already there" guards then leave our correctly-coded fields alone. No change to AddAuthority() itself is needed for this. This is MARC21-only (matches the existing marcflavour branch already in this code path) and ships as an unconditional bug fix, with no new system preference: it only changes how a *new* authority is coded going forward and does not touch or merge any existing authority records. Test plan (librarian-facing): 1. Turn on the AutoLinkBiblios and AutoCreateAuthorities system preferences (Administration -> System preferences -> Cataloging). 2. Catalog a new bibliographic record, or edit an existing one, and add a 650 field coded for a non-LCSH thesaurus, e.g.: 650 _7 $a Miracles $2 fast (2nd indicator = 7, subfield $2 = fast; a FAST heading with no existing matching authority in your catalogue). 3. Save the record. Koha auto-creates a new "Miracles" authority record, since none existed yet to link to. 4. In the Authorities module, open the newly created "Miracles" record for editing and switch to MARC view (Edit -> Edit record, or the "..." menu -> Edit as new). 5. Click into the 008 field to open its editor plugin. Before this fix: the "11- Subject heading system/thesaurus" dropdown is set to "a- Library of Congress Subject Headings", and field 040 has no $f subfield. After this fix: that dropdown is now set to "z- Other", and field 040 has a $f subfield containing "fast" — correctly recording the real source vocabulary instead of silently mis-labelling it as LCSH. 6. As a regression check, repeat steps 2-5 with a plain LCSH heading instead, e.g.: 650 __0 $a Feminism (2nd indicator = 0, no $2). Confirm the resulting authority's 008 still shows "a- Library of Congress Subject Headings" as before — this fix must not change behaviour for ordinary LCSH headings. 7. Optional, Elasticsearch installations only: with LinkerConsiderThesaurus also turned on, re-import or re-save a bib with the same FAST heading from step 2 a second time. It should now link to the authority created in step 3 instead of creating a second duplicate — this is the actual duplicate-authority problem reported in bug 42694. Sponsored-by: Heythrop Library <https://heythroplibrary.co.uk/> Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 --- Comment #11 from Phil Ringnalda <phil@chetcolibrary.org> --- Created attachment 201872 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201872&action=edit Bug 31925: Make QueryBuilder reuse the shared thesaurus/008 lookup Koha::SearchEngine::Elasticsearch::QueryBuilder kept its own copy of the thesaurus-name to 008/11-code table used when building "thesaurus" search queries against authority records. The previous commit moved that same table to Koha::Authority as a shared lookup. This commit deletes QueryBuilder's duplicate and points $thesaurus_to_value at $Koha::Authority::MARC21_THESAURUS_TO_CONTROL_FIELD_008_11 instead, so there is only one place to update if the code list ever changes. Pure refactor: $thesaurus_to_value keeps its existing name and contents, and build_authorities_query_compat() is unchanged, so query behaviour is identical before and after. Sponsored-by: Heythrop Library <https://heythroplibrary.co.uk/> Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lisette@bywatersolutions.co | |m QA Contact|testopia@bugs.koha-communit |lisette@bywatersolutions.co |y.org |m -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 --- Comment #13 from cgresser <cgresser@jesuit.org.uk> --- I'm not sure whether it's custom to do this, but we have been testing the patch for this bug since yesterday, and it works like a charm: 650 _ 0 ‡aChildbirth‡xMoral and ethical aspects.‡9217329 650 _ 7 ‡aChildbirth‡xMoral and ethical aspects‡2fast‡9217330 Both were new subject headings / access points for us, and they were promptly created by Koha for our local instance/local authorities database. The following is an example of an existing standard LCC heading, and then Koha picked up that the fast one needs to be added to our local authorities: 650 _ 0 ‡aPregnancy‡xMoral and ethical aspects.‡9135277 650 _ 7 ‡aPregnancy‡xMoral and ethical aspects‡2fast‡9217331 It's like Christmas has come early! Now to fixing our local heritage authorities data, which is not Koha's fault. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 --- Comment #15 from David Cook <dcook@prosentient.com.au> --- Yeah... I'm slowly starting to tease apart these changes in my mind. I find the Linker and C4::Biblio::LinkBibHeadingsToAuthorities to be quite... interesting... every time I look at them. Ok so LinkBibHeadingsToAuthorities iterates through the bib fields... and using C4::Heading->new_from_field, we get a "heading" object if the MARC Bibliographic Framework is linked to a "Thesaurus" (ie Authority Type), which is problematic in many cases because of the hard-coded codes that bug 41714 tries to solve... The bib field (hard-coded to 6XX) has a 2nd indicator which indicates which thesaurus the subject heading belongs to... Authority records have a position in their 008 which indicates which thesaurus they're from. Got it... -- Ok, the comments and code look more complicated than they are then. A certain number of the changes are actually just refactoring to make the code DRYer... So basically... what we're trying to say here is "AutoCreateAuthorities must include the thesaurus from the bib heading when auto creating a new authority record". Cool. Easy. I reckon we should include more code comments, because it's often not obvious what the code is doing... -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 --- Comment #17 from David Cook <dcook@prosentient.com.au> --- Created attachment 204569 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=204569&action=edit Bug 31925: Add thesaurus/008 lookup and default MARC21 008 builder to Koha::Authority AutoCreateAuthorities builds a brand-new authority record whenever a bib subject heading (6XX) has no match, but always lets the record fall back to C4::AuthoritiesMarc::AddAuthority()'s default 008/040, which hardcodes 008 position 11 ("Subject heading system/thesaurus") to 'a' (LCSH) no matter what thesaurus the source heading actually uses. This is the first step towards fixing that. Adds two things to Koha::Authority: - $MARC21_THESAURUS_TO_CONTROL_FIELD_008_11: a single-source-of-truth lookup mapping a thesaurus name (as already derived by C4::Heading from MARC indicator 2 / subfield $2, e.g. 'lcsh', 'mesh', or a raw $2 code such as 'fast') to its MARC21 authority 008/11 code, per the LOC ad008 code list. This table previously only existed once, for a different purpose, in Koha::SearchEngine::Elasticsearch::QueryBuilder (used for building thesaurus search queries) - it's promoted here so both that and the authority-creation fix that follows can read from one place instead of two copies that can silently drift apart. - default_marc21_008($thesaurus): builds the default 34-character body of a new MARC21 authority's 008 field, honouring the MARCAuthorityControlField008 system preference. When $thesaurus is given, position 11 is set from the table above instead of whatever the default/syspref value carries, falling back to 'z' (Other) for a thesaurus with no dedicated code. AddAuthority() is refactored to build its own (thesaurus-less, LCSH default) 008 via this new method too, replacing its previous inline copy of the same defaulting logic - pure refactor, no behaviour change. Nothing thesaurus-aware calls it yet. Sponsored-by: Heythrop Library <https://heythroplibrary.co.uk/> Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org> Signed-off-by: David Cook <dcook@prosentient.com.au> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 --- Comment #19 from David Cook <dcook@prosentient.com.au> --- Created attachment 204571 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=204571&action=edit Bug 31925: AutoCreateAuthorities must preserve subject heading thesaurus When AutoCreateAuthorities creates a new authority for a bib subject heading that didn't match an existing one, it never set 008 or 040 on the new MARC::Record, so C4::AuthoritiesMarc::AddAuthority()'s own defaulting always ran and hardcoded 008/11 to 'a' (LCSH), regardless of the source heading's real thesaurus. A FAST, MeSH, or other non-LCSH heading (MARC indicator 2 = 7 with subfield $2, or indicator 2 = 1-6) therefore always got mis-coded as LCSH on creation. Because the authority was permanently mis-coded, it could never be found again by thesaurus-aware matching (LinkerConsiderThesaurus), so every later import of the same non-LCSH heading created yet another duplicate authority instead of linking to the one already created (see bug 42694 for a real-world report of this). This calls Koha::Authority->default_marc21_008($thesaurus) to build the new authority's 008 - correctly coding position 11 for the source heading's actual thesaurus - and, for headings identified only by a raw $2 code, adds a 040$f before calling AddAuthority(), whose existing "only default if the field isn't already there" guards then leave our correctly-coded fields alone. No change to AddAuthority() itself is needed for this. This is MARC21-only (matches the existing marcflavour branch already in this code path) and ships as an unconditional bug fix, with no new system preference: it only changes how a *new* authority is coded going forward and does not touch or merge any existing authority records. Test plan (librarian-facing): 1. Turn on the AutoLinkBiblios and AutoCreateAuthorities system preferences (Administration -> System preferences -> Cataloging). 2. Catalog a new bibliographic record, or edit an existing one, and add a 650 field coded for a non-LCSH thesaurus, e.g.: 650 _7 $a Miracles $2 fast (2nd indicator = 7, subfield $2 = fast; a FAST heading with no existing matching authority in your catalogue). 3. Save the record. Koha auto-creates a new "Miracles" authority record, since none existed yet to link to. 4. In the Authorities module, open the newly created "Miracles" record for editing and switch to MARC view (Edit -> Edit record, or the "..." menu -> Edit as new). 5. Click into the 008 field to open its editor plugin. Before this fix: the "11- Subject heading system/thesaurus" dropdown is set to "a- Library of Congress Subject Headings", and field 040 has no $f subfield. After this fix: that dropdown is now set to "z- Other", and field 040 has a $f subfield containing "fast" — correctly recording the real source vocabulary instead of silently mis-labelling it as LCSH. 6. As a regression check, repeat steps 2-5 with a plain LCSH heading instead, e.g.: 650 __0 $a Feminism (2nd indicator = 0, no $2). Confirm the resulting authority's 008 still shows "a- Library of Congress Subject Headings" as before — this fix must not change behaviour for ordinary LCSH headings. 7. Optional, Elasticsearch installations only: with LinkerConsiderThesaurus also turned on, re-import or re-save a bib with the same FAST heading from step 2 a second time. It should now link to the authority created in step 3 instead of creating a second duplicate — this is the actual duplicate-authority problem reported in bug 42694. Sponsored-by: Heythrop Library <https://heythroplibrary.co.uk/> Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org> Signed-off-by: David Cook <dcook@prosentient.com.au> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 --- Comment #21 from David Cook <dcook@prosentient.com.au> --- Doesn't apply cleanly to 25.11, but the merge conflicts were pretty easy to fix. I think the conflicts are caused by bug 42032 so 26.05 would probably be fine -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 --- Comment #23 from David Cook <dcook@prosentient.com.au> --- Hmm I just noticed the fallback to "z" for position 11 in the authority 008... That's going to be a change of behaviour which I could see causing more bugs than it fixes... -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Signed Off --- Comment #25 from David Cook <dcook@prosentient.com.au> --- Ok, so if you try to save a 650 without a value in ind2, you'll get a C4::Heading object with a "thesaurus" property of "notdefined". To me, "notdefined" is the same as "notspecified". According to https://www.loc.gov/marc/authority/ad008.html, "z" for position 11 is "Other". In this case I think "| - No attempt to code" is more appropriate. Going to move this back to Signed Off for now while we discuss... -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 --- Comment #27 from Phil Ringnalda <phil@chetcolibrary.org> --- (In reply to David Cook from comment #25)

Ok, so if you try to save a 650 without a value in ind2, you'll get a C4::Heading object with a "thesaurus" property of "notdefined".

And will another bib heading with no ind2 be linked to the heading created with 008/11 = 'z'? Keep in mind, that's the entire point of this bug, to create headings which will match what the linker looks for. Whether a blank indicator in a bib record should link to 'z' or '|' would be a followup bug that blocks bug 30280, which has been linking to 'z' since 22.11, which is a fair amount of existing behavior to buck. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 --- Comment #29 from David Cook <dcook@prosentient.com.au> --- (In reply to David Cook from comment #28)

Bug 31925, as it sits right now, would likely be very disruptive.

Note also that these patches would also break the behaviour of the "MARCAuthorityControlField008" system preference for position 008/11. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Signed Off -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 --- Comment #31 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 204579 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=204579&action=edit Bug 31925: Add thesaurus/008 lookup and default MARC21 008 builder to Koha::Authority AutoCreateAuthorities builds a brand-new authority record whenever a bib subject heading (6XX) has no match, but always lets the record fall back to C4::AuthoritiesMarc::AddAuthority()'s default 008/040, which hardcodes 008 position 11 ("Subject heading system/thesaurus") to 'a' (LCSH) no matter what thesaurus the source heading actually uses. This is the first step towards fixing that. Adds two things to Koha::Authority: - $MARC21_THESAURUS_TO_CONTROL_FIELD_008_11: a single-source-of-truth lookup mapping a thesaurus name (as already derived by C4::Heading from MARC indicator 2 / subfield $2, e.g. 'lcsh', 'mesh', or a raw $2 code such as 'fast') to its MARC21 authority 008/11 code, per the LOC ad008 code list. This table previously only existed once, for a different purpose, in Koha::SearchEngine::Elasticsearch::QueryBuilder (used for building thesaurus search queries) - it's promoted here so both that and the authority-creation fix that follows can read from one place instead of two copies that can silently drift apart. - default_marc21_008($thesaurus): builds the default 34-character body of a new MARC21 authority's 008 field, honouring the MARCAuthorityControlField008 system preference. When $thesaurus is given, position 11 is set from the table above instead of whatever the default/syspref value carries, falling back to 'z' (Other) for a thesaurus with no dedicated code. AddAuthority() is refactored to build its own (thesaurus-less, LCSH default) 008 via this new method too, replacing its previous inline copy of the same defaulting logic - pure refactor, no behaviour change. Nothing thesaurus-aware calls it yet. Sponsored-by: Heythrop Library <https://heythroplibrary.co.uk/> Signed-off-by: Clemens Gresser <cgresser@jesuit.org.uk> Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org> Signed-off-by: David Cook <dcook@prosentient.com.au> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 --- Comment #33 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 204581 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=204581&action=edit Bug 31925: AutoCreateAuthorities must preserve subject heading thesaurus When AutoCreateAuthorities creates a new authority for a bib subject heading that didn't match an existing one, it never set 008 or 040 on the new MARC::Record, so C4::AuthoritiesMarc::AddAuthority()'s own defaulting always ran and hardcoded 008/11 to 'a' (LCSH), regardless of the source heading's real thesaurus. A FAST, MeSH, or other non-LCSH heading (MARC indicator 2 = 7 with subfield $2, or indicator 2 = 1-6) therefore always got mis-coded as LCSH on creation. Because the authority was permanently mis-coded, it could never be found again by thesaurus-aware matching (LinkerConsiderThesaurus), so every later import of the same non-LCSH heading created yet another duplicate authority instead of linking to the one already created (see bug 42694 for a real-world report of this). This calls Koha::Authority->default_marc21_008($thesaurus) to build the new authority's 008 - correctly coding position 11 for the source heading's actual thesaurus - and, for headings identified only by a raw $2 code, adds a 040$f before calling AddAuthority(), whose existing "only default if the field isn't already there" guards then leave our correctly-coded fields alone. No change to AddAuthority() itself is needed for this. This is MARC21-only (matches the existing marcflavour branch already in this code path) and ships as an unconditional bug fix, with no new system preference: it only changes how a *new* authority is coded going forward and does not touch or merge any existing authority records. Test plan (librarian-facing): 1. Turn on the AutoLinkBiblios and AutoCreateAuthorities system preferences (Administration -> System preferences -> Cataloging). 2. Catalog a new bibliographic record, or edit an existing one, and add a 650 field coded for a non-LCSH thesaurus, e.g.: 650 _7 $a Miracles $2 fast (2nd indicator = 7, subfield $2 = fast; a FAST heading with no existing matching authority in your catalogue). 3. Save the record. Koha auto-creates a new "Miracles" authority record, since none existed yet to link to. 4. In the Authorities module, open the newly created "Miracles" record for editing and switch to MARC view (Edit -> Edit record, or the "..." menu -> Edit as new). 5. Click into the 008 field to open its editor plugin. Before this fix: the "11- Subject heading system/thesaurus" dropdown is set to "a- Library of Congress Subject Headings", and field 040 has no $f subfield. After this fix: that dropdown is now set to "z- Other", and field 040 has a $f subfield containing "fast" — correctly recording the real source vocabulary instead of silently mis-labelling it as LCSH. 6. As a regression check, repeat steps 2-5 with a plain LCSH heading instead, e.g.: 650 __0 $a Feminism (2nd indicator = 0, no $2). Confirm the resulting authority's 008 still shows "a- Library of Congress Subject Headings" as before — this fix must not change behaviour for ordinary LCSH headings. 7. Optional, Elasticsearch installations only: with LinkerConsiderThesaurus also turned on, re-import or re-save a bib with the same FAST heading from step 2 a second time. It should now link to the authority created in step 3 instead of creating a second duplicate — this is the actual duplicate-authority problem reported in bug 42694. Sponsored-by: Heythrop Library <https://heythroplibrary.co.uk/> Signed-off-by: Clemens Gresser <cgresser@jesuit.org.uk> Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org> Signed-off-by: David Cook <dcook@prosentient.com.au> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Target Milestone|--- |26.11 Initiative type|--- |Feature -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=43483 -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 Esther Melander <esther.melander@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |esther.melander@bywatersolu | |tions.com -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 --- Comment #37 from Phil Ringnalda <phil@chetcolibrary.org> --- FWIW, the more I try to put myself in the place of someone who doesn't want to turn on LinkerConsiderThesaurus the more I think that the right move is to instead only autocreate authorities with the correct thesaurus when LinkerConsiderThesaurus is on, and to continue creating them based only on whatever is in MARCAuthorityControlField008 when it is off. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 --- Comment #39 from David Cook <dcook@prosentient.com.au> --- Created attachment 205270 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205270&action=edit Bug 31925: Add thesaurus/008 lookup and default MARC21 008 builder to Koha::Authority AutoCreateAuthorities builds a brand-new authority record whenever a bib subject heading (6XX) has no match, but always lets the record fall back to C4::AuthoritiesMarc::AddAuthority()'s default 008/040, which hardcodes 008 position 11 ("Subject heading system/thesaurus") to 'a' (LCSH) no matter what thesaurus the source heading actually uses. This is the first step towards fixing that. Adds two things to Koha::Authority: - $MARC21_THESAURUS_TO_CONTROL_FIELD_008_11: a single-source-of-truth lookup mapping a thesaurus name (as already derived by C4::Heading from MARC indicator 2 / subfield $2, e.g. 'lcsh', 'mesh', or a raw $2 code such as 'fast') to its MARC21 authority 008/11 code, per the LOC ad008 code list. This table previously only existed once, for a different purpose, in Koha::SearchEngine::Elasticsearch::QueryBuilder (used for building thesaurus search queries) - it's promoted here so both that and the authority-creation fix that follows can read from one place instead of two copies that can silently drift apart. - default_marc21_008($thesaurus): builds the default 34-character body of a new MARC21 authority's 008 field, honouring the MARCAuthorityControlField008 system preference. When $thesaurus is given, position 11 is set from the table above instead of whatever the default/syspref value carries, falling back to 'z' (Other) for a thesaurus with no dedicated code. AddAuthority() is refactored to build its own (thesaurus-less, LCSH default) 008 via this new method too, replacing its previous inline copy of the same defaulting logic - pure refactor, no behaviour change. Nothing thesaurus-aware calls it yet. Sponsored-by: Heythrop Library <https://heythroplibrary.co.uk/> Signed-off-by: Clemens Gresser <cgresser@jesuit.org.uk> Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org> Signed-off-by: David Cook <dcook@prosentient.com.au> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 --- Comment #41 from David Cook <dcook@prosentient.com.au> --- Created attachment 205272 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205272&action=edit Bug 31925: AutoCreateAuthorities must preserve subject heading thesaurus When AutoCreateAuthorities creates a new authority for a bib subject heading that didn't match an existing one, it never set 008 or 040 on the new MARC::Record, so C4::AuthoritiesMarc::AddAuthority()'s own defaulting always ran and hardcoded 008/11 to 'a' (LCSH), regardless of the source heading's real thesaurus. A FAST, MeSH, or other non-LCSH heading (MARC indicator 2 = 7 with subfield $2, or indicator 2 = 1-6) therefore always got mis-coded as LCSH on creation. Because the authority was permanently mis-coded, it could never be found again by thesaurus-aware matching (LinkerConsiderThesaurus), so every later import of the same non-LCSH heading created yet another duplicate authority instead of linking to the one already created (see bug 42694 for a real-world report of this). This calls Koha::Authority->default_marc21_008($thesaurus) to build the new authority's 008 - correctly coding position 11 for the source heading's actual thesaurus - and, for headings identified only by a raw $2 code, adds a 040$f before calling AddAuthority(), whose existing "only default if the field isn't already there" guards then leave our correctly-coded fields alone. No change to AddAuthority() itself is needed for this. This is MARC21-only (matches the existing marcflavour branch already in this code path) and ships as an unconditional bug fix, with no new system preference: it only changes how a *new* authority is coded going forward and does not touch or merge any existing authority records. Test plan (librarian-facing): 1. Turn on the AutoLinkBiblios and AutoCreateAuthorities system preferences (Administration -> System preferences -> Cataloging). 2. Catalog a new bibliographic record, or edit an existing one, and add a 650 field coded for a non-LCSH thesaurus, e.g.: 650 _7 $a Miracles $2 fast (2nd indicator = 7, subfield $2 = fast; a FAST heading with no existing matching authority in your catalogue). 3. Save the record. Koha auto-creates a new "Miracles" authority record, since none existed yet to link to. 4. In the Authorities module, open the newly created "Miracles" record for editing and switch to MARC view (Edit -> Edit record, or the "..." menu -> Edit as new). 5. Click into the 008 field to open its editor plugin. Before this fix: the "11- Subject heading system/thesaurus" dropdown is set to "a- Library of Congress Subject Headings", and field 040 has no $f subfield. After this fix: that dropdown is now set to "z- Other", and field 040 has a $f subfield containing "fast" — correctly recording the real source vocabulary instead of silently mis-labelling it as LCSH. 6. As a regression check, repeat steps 2-5 with a plain LCSH heading instead, e.g.: 650 __0 $a Feminism (2nd indicator = 0, no $2). Confirm the resulting authority's 008 still shows "a- Library of Congress Subject Headings" as before — this fix must not change behaviour for ordinary LCSH headings. 7. Optional, Elasticsearch installations only: with LinkerConsiderThesaurus also turned on, re-import or re-save a bib with the same FAST heading from step 2 a second time. It should now link to the authority created in step 3 instead of creating a second duplicate — this is the actual duplicate-authority problem reported in bug 42694. Sponsored-by: Heythrop Library <https://heythroplibrary.co.uk/> Signed-off-by: Clemens Gresser <cgresser@jesuit.org.uk> Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org> Signed-off-by: David Cook <dcook@prosentient.com.au> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 --- Comment #43 from Phil Ringnalda <phil@chetcolibrary.org> --- Import a bib record with 650_7$aDogs.$2fast which autocreates a Fast authority record, then import a thousand bib records with 650_0$aDogs. which will link to that authority record, then innocently save the authority record which will change your thousand bib records from LC to Fast. As a cataloger, I would not consider that harmless. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |43483 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43483 [Bug 43483] Authority linker treats the same name heading differently in 6XX vs 1XX/7XX -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 --- Comment #46 from David Cook <dcook@prosentient.com.au> --- (In reply to Pedro Amorim (ammopt) from comment #45)

Hi guys, confirming Phil's concern from comment 43:

1) Turn on AutoLinkBiblios and AutoCreateAuthorities (leave LinkerConsiderThesaurus off, that's the default). 2) Catalog a new bib record. Give it any title. Add a 650 field: 2nd indicator = 7, subfield a = Zzztestthesaurus, subfield 2 = fast. Save it. 3) Catalog a second new bib record. Give it any title. Add a 650 field: 2nd indicator = blank, subfield a = Zzztestthesaurus, no $2. Save it. 4) Search Authorities for Zzztestthesaurus. There should be exactly one result, meaning both bibs linked to the same authority. 5) Open that authority record and just save it again with no changes. 6) Reopen the second bib record and look at its 650 field. => If the 2nd indicator is now 7 and a $2 fast has appeared, even though you typed neither, that's the bug.

Yeah, no I understand the issue, but that's not unique to this change. I don't think it's relevant. Consider the following: 0) Checkout "main" 1) Turn on AutoLinkBiblios and AutoCreateAuthorities (leave LinkerConsiderThesaurus off, that's the default). 2) Go to /cgi-bin/koha/authorities/authorities.pl?authtypecode=TOPIC_TERM to create a new TOPIC TERM 3) In the 008, change "11-Subject heading system/thesaurus" to "c- Medical subject headings" 4) Change the 150$a to "AwesomeTopic" 5) Create a new bib record with 650$a AwesomeTopic (but don't link it) 6) After saving, review the record and notice that it's been AutoLinked to your AwesomeTopic authority but the indicators are still empty 7) Go back to your authority record (e.g. /cgi-bin/koha/authorities/detail.pl?authid=1730), open it in the editor, click Save 8) Go back to your bib record and note that the 2nd indicator for 650$a AwesomeTopic now includes "2"! So do you see how that updating of bibs from the authority is the same regardless of *how* the authority was created? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 --- Comment #48 from Phil Ringnalda <phil@chetcolibrary.org> --- I didn't bring it up because I thought it was new behavior. I brought it up because it's the existing behavior which will impact workflows as a result of this patch. For as long as AutoCreateAuthorities has existed, it has created them ignoring the thesaurus in the bib record. If as one of my fellow catalogers does, you save a bib imported from OCLC and then delete all the headings in it from thesauri we don't use, then no matter which flavor of a heading came first our current install of Koha will create an LC authority record from it, and saving the authority record will at most (if some of the garbage wasn't deleted) change the thesaurus from one we don't use to the one we do use. If this change applies to installs that don't use LinkerConsiderThesaurus as well as those that do, then those installs will have to notice that this bug landed, realize that it will be creating authorities from a thesaurus they don't want, and alter their system to always check the thesaurus for any newly created authority record before touching it. That would leave me having to remember to *always* switch to a different tab, and open up a popup window to be sure that I was looking at the correct space in a multicharacter string for an 'a' to be sure something else hadn't slipped in. I mark ones that I've checked, so I would only have to do that once, but for people who don't, they would need to do that every time. Or work out a way to find every authority record not in the thesaurus they want, and batch change them all, before any time they do any work that might result in saving an authority record. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 --- Comment #50 from Phil Ringnalda <phil@chetcolibrary.org> --- My workflow will be to stay up until 1 or 2am the night this patch hits my production system, to toggle LinkerConsiderThesaurus to 'Do' as soon as I can. I've been waiting to be able to do that for four years now. What I'm trying to do is think about what someone who is choosing not to do that would prefer here. Since I'm apparently not good at communicating why it's bad, let's flip it on its head. Why do you insist that it is better to create an authority record for whatever thesaurus happens to be the first one seen by AutoCreateAuthorities in a system where LinkerConsiderThesaurus is turned off? If they will never even notice, why do it? Is "whatever bib heading happens to be seen first" actually what someone who has a particular thesaurus in their MARCAuthorityControlField008 preference really wants? If they want to have authority records reflect the thesaurus, why aren't they turning on LinkerConsiderThesaurus? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 --- Comment #52 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 205403 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205403&action=edit Bug 31925: Only code auto-created authorities with the source thesaurus when LinkerConsiderThesaurus is on Comments 43-51 identified a further problem with unconditionally coding a newly auto-created authority's 008/11 (and 040$f) from the source bib heading's thesaurus: when LinkerConsiderThesaurus is off (the default), the linker matches new headings to existing authorities by text alone, regardless of thesaurus. So a bib heading from one thesaurus (e.g. FAST) can auto-create an authority coded for that thesaurus, and a later bib heading with the *same* text but no thesaurus of its own (or a different one) can then link to that same authority purely on the text match. Saving that authority afterwards propagates its 008/11 and 040$f coding back onto every bib heading linked to it - silently adding indicators and subfields the cataloguer never set, on installs that never opted into thesaurus-aware behaviour. Before this bug, AutoCreateAuthorities always created LCSH-coded records, so that propagation was always onto the coding most sites already expect. This bug's earlier patches changed that default for every site regardless of whether they use LinkerConsiderThesaurus, which is what comments 43/45 (reproduced by Pedro Amorim) demonstrated as disruptive. Sites that haven't turned on LinkerConsiderThesaurus don't benefit from thesaurus-aware authority creation anyway, since their linker never searches by thesaurus - C4::Heading::_search and C4::Linker::Default already gate thesaurus-consideration on this same preference for matching. This change gates authority-creation coding on it too, so the behaviour introduced by this bug only applies to installs that already opted into thesaurus-aware linking. Test plan: 1. prove t/db_dependent/Biblio.t Note the new case 7 in the "codes new authorities with the source thesaurus" subtest: with LinkerConsiderThesaurus off, a FAST heading no longer codes the new authority as FAST, it keeps the MARCAuthorityControlField008 syspref default instead. 2. Librarian-facing regression check: with LinkerConsiderThesaurus off (the default), AutoLinkBiblios and AutoCreateAuthorities on, catalogue a 650 with ind2=7 $2=fast and save. The new authority's 008/11 should be whatever MARCAuthorityControlField008 says (LCSH 'a' by default), not 'z'. 3. Confirm the LinkerConsiderThesaurus=on cases from the earlier patches are unaffected: with it on, a FAST heading still codes 008/11='z' with 040$f=fast, and a MeSH heading still codes 008/11='c'. Sponsored-by: Heythrop Library <https://heythroplibrary.co.uk/> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Signed Off -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 Phil Ringnalda <phil@chetcolibrary.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #205403|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 --- Comment #56 from Phil Ringnalda <phil@chetcolibrary.org> --- It's lovely, and I look forward to running it. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Failed QA --- Comment #58 from David Cook <dcook@prosentient.com.au> --- Actually, I don't have time for this today... sorry. Going to mark as Failed QA. Might be worth wrapping it with a SearchEngine eq Elasticsearch too... because with Zebra it just creates duplicate authorities... -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31925 --- Comment #60 from David Cook <dcook@prosentient.com.au> --- (In reply to Phil Ringnalda from comment #59)

There was a time when it didn't work on Zebra, but I think that's been fixed since attachment 148416 [details] [review] from bug 33159. I just tried this patch with Zebra, and everything seemed to work correctly.

I tried it with Zebra and I'm getting duplicate authorities when I have LinkerConsiderThesaurus enabled, which makes sense since LinkerConsiderThesaurus is an Elastic-only feature. -- You are receiving this mail because: You are watching all bug changes.