[Bug 24094] New: Authority punctuation mismatch prevents linking to correct records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24094 Bug ID: 24094 Summary: Authority punctuation mismatch prevents linking to correct records Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: trivial Priority: P5 - low Component: MARC Authority data support Assignee: koha-bugs@lists.koha-community.org Reporter: bwsdonna@gmail.com QA Contact: testopia@bugs.koha-community.org In a bibliographic record, the comma preceding the relationship designator, i.e., the comma before $e, is prescribed--it has to be there for the record to conform to RDA and MARC21. Authority records view this as a different name/etc, so it does not match. Every, single, correctly cataloged RDA/MARC21 record is going to have a $e preceded by a comma in every authority controlled X00 and X10 field. Authorities need to disregard/strip trailing commas. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24094 Myka Kennedy Stephens <mkstephens@lancasterseminary.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mkstephens@lancasterseminar | |y.edu -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24094 --- Comment #1 from Myka Kennedy Stephens <mkstephens@lancasterseminary.edu> --- Donna, Would you clarify which search engine you have enabled when you're seeing this behavior? I am doing some extensive testing with authorities on Elasticsearch and am not seeing this problem with the linker. I also have broader_headings enabled under LinkerOptions. My experience is that the linker is correctly matching x00 headings with a relator term present *only* when the x00 heading has only $a and $e present. For headings that also include dates ($d) or any other subfields, the linker will not find a match. I'm thinking this is a different bug and will be submitting it after I do some more tests. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24094 --- Comment #2 from Donna <bwsdonna@gmail.com> --- (In reply to Myka Kennedy Stephens from comment #1)
Donna, Would you clarify which search engine you have enabled when you're seeing this behavior? I am doing some extensive testing with authorities on Elasticsearch and am not seeing this problem with the linker. I also have broader_headings enabled under LinkerOptions. My experience is that the linker is correctly matching x00 headings with a relator term present *only* when the x00 heading has only $a and $e present. For headings that also include dates ($d) or any other subfields, the linker will not find a match. I'm thinking this is a different bug and will be submitting it after I do some more tests.
This is on Zebra, Myka. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24094 Heather <heather_hernandez@nps.gov> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |heather_hernandez@nps.gov -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24094 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|trivial |normal CC| |katrin.fischer@bsz-bw.de -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24094 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24094 --- Comment #3 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 98238 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=98238&action=edit Bug 24094: Unit tests -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24094 --- Comment #4 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 98239 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=98239&action=edit Bug 24094: Strip trailing spaces and punctuation from authority headings Both when searching for and creating new authorities we need to remove punctuation that exists in the bibliographic record but does not belong in the authority record. For example, a series with a volume contains a semicolon in the bib record, however, this should not be passed to the authority as the volume is not included in the authority record. To test: 1 - Set AutoCreateAuthorities to 'generate' 2 - Set BiblioAddsAuthorities to 'true' 3 - Set CatalogModuleRelink to 'Do' 4 - Find or create a record with: a 100 field with a subfield e preceded by a comma: 100 $aBoring, M. Eugene M ,$e author an 830 field with a volume preceded by a semicolon: 650$aLord of the rings ;$v 3. 5 - Save the records and check the links 6 - Note punctuation is passed through 7 - Save again, auth records are created again 8 - Apply patch 9 - Save again, new auth records are created again 10 - Check the records, punctuation has been removed 11 - Save again, no more records created. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24094 Myka Kennedy Stephens <mkstephens@lancasterseminary.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA --- Comment #5 from Myka Kennedy Stephens <mkstephens@lancasterseminary.edu> --- These patches work as expected in both Zebra and ES for personal names. Applied the patch and saved the record again. It stripped the ending "," before $e in a 100 field and created a new authority record. Re-saving the record again kept the 100 field linked to the same authority record. Uniform titles are another matter. Applied patch and saved the record again. It stripped the ending ";" before $v in an 830 field and created a new authority record. Re-saving the record again created another new authority record and re-linked the 830 to the new authority record. I'm also noticing that it is still creating additional authority records for other fields, but not consistently. For example, one record I tried kept subject heading links between saves, but another record did not. Switching between Zebra and ES didn't seem to make a difference. The bibliographic records I tested came from the Library of Congress Z39.50 server: #1 - ISBN 9781540960573 #2 - ISBN 9789004130784 #3 - ISBN 9781532653308 -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24094 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |nick@bywatersolutions.com |ity.org | CC| |nick@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24094 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff --- Comment #6 from Nick Clemens <nick@bywatersolutions.com> --- (In reply to Myka Kennedy Stephens from comment #5)
The bibliographic records I tested came from the Library of Congress Z39.50 server: #1 - ISBN 9781540960573 #2 - ISBN 9789004130784 #3 - ISBN 9781532653308
I applied the patches, imported the three records, and saved multiple times - the authorities were generated once, and didn't happen again, everything correctly relinked. There is the possibility that your linker was set to 'Default' and so not linking because there were duplicates in the system and the Default linker only links if a single record is returned. Can you retest with 'First' or 'Last' match linker, and or clearing out the duplicates? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24094 Myka Kennedy Stephens <mkstephens@lancasterseminary.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24094 Myka Kennedy Stephens <mkstephens@lancasterseminary.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #98238|0 |1 is obsolete| | --- Comment #7 from Myka Kennedy Stephens <mkstephens@lancasterseminary.edu> --- Created attachment 98735 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=98735&action=edit Bug 24094: Unit tests Signed-off-by: Myka Kennedy Stephens <mkstephens@lancasterseminary.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24094 Myka Kennedy Stephens <mkstephens@lancasterseminary.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #98239|0 |1 is obsolete| | --- Comment #8 from Myka Kennedy Stephens <mkstephens@lancasterseminary.edu> --- Created attachment 98736 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=98736&action=edit Bug 24094: Strip trailing spaces and punctuation from authority headings Both when searching for and creating new authorities we need to remove punctuation that exists in the bibliographic record but does not belong in the authority record. For example, a series with a volume contains a semicolon in the bib record, however, this should not be passed to the authority as the volume is not included in the authority record. To test: 1 - Set AutoCreateAuthorities to 'generate' 2 - Set BiblioAddsAuthorities to 'true' 3 - Set CatalogModuleRelink to 'Do' 4 - Find or create a record with: a 100 field with a subfield e preceded by a comma: 100 $aBoring, M. Eugene M ,$e author an 830 field with a volume preceded by a semicolon: 650$aLord of the rings ;$v 3. 5 - Save the records and check the links 6 - Note punctuation is passed through 7 - Save again, auth records are created again 8 - Apply patch 9 - Save again, new auth records are created again 10 - Check the records, punctuation has been removed 11 - Save again, no more records created. Signed-off-by: Myka Kennedy Stephens <mkstephens@lancasterseminary.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24094 --- Comment #9 from Myka Kennedy Stephens <mkstephens@lancasterseminary.edu> --- (In reply to Myka Kennedy Stephens from comment #8)
To test: 1 - Set AutoCreateAuthorities to 'generate' 2 - Set BiblioAddsAuthorities to 'true' The setting is actually 'allow' 3 - Set CatalogModuleRelink to 'Do' 3a - Set LinkerModule to 'First Match' 4 - Find or create a record with: a 100 field with a subfield e preceded by a comma: 100 $aBoring, M. Eugene M ,$e author an 830 field with a volume preceded by a semicolon: 650$aLord of the rings ;$v 3. 5 - Save the records and check the links 6 - Note punctuation is passed through 7 - Save again, auth records are created again 8 - Apply patch 9 - Save again, new auth records are created again 10 - Check the records, punctuation has been removed 11 - Save again, no more records created.
[HUGE] CAVEATS - 1. This patch works optimally with Zebra. Results are mixed with ES. It is looking like ES cannot match headings with any type of punctuation--parentheses, hyphens, periods within headings, etc. Further development work will be needed to address this problem with ES. To observe the behavior, test with ES enabled and import/edit ISBN 9789004130784 which includes several subject headings with internal punctuation and a series title heading with parentheses. 2. This patch will strip any trailing punctuation. This includes a period after a middle initial on a personal name heading if that period is not followed by any other punctuation. It would also include the period at the end of a subject heading that ends with etc., B.C., or any other abbreviation in which a period might need to be preserved. This feels like a bandage that would hold for a little while but isn't a long-term fix. Bug #11300 holds a lot of promise to be a long-term fix, especially if the linker could search a Z39.50 server for a record during the cataloging process and import it (or prompt to import) if it doesn't exist. Ideally, this would import the record with the correct punctuation for an authority record while also preserving the punctuation in the bib record. In sum, I'm signing off on this but recognize that it has limitations. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24094 --- Comment #10 from Myka Kennedy Stephens <mkstephens@lancasterseminary.edu> --- Follow-up and further thoughts: Re: Caveat #1 - I went back and tried this again after making sure that the ES authority mapping on my test box was correct and complete. The linker still failed to link the headings. I am now wondering if bug #24269 could be the root cause of that, rather than a shortcoming on the patch for this bug. I tried to apply the patch for 24269 on top of the patch for this bug, but it failed to apply (I know it has already been signed off, though). Re: Caveat #2 - *If* one wanted to build a list of exceptions of authority punctuation that would need to be kept, here are a few places to start (from an incomplete U.S./Library of Congress perspective): - etc. - B.C. - A.D. - a period following a single letter (as in an initial) - Dr. - Jr. - Mrs. - Co. - Dept. - Inc. - hyphen (-) - closing parenthesis ) - question mark ? - exclamation point ! If this is the path taken, I strongly recommend this list be expanded to be inclusive of punctuation and abbreviations used in authority records of languages other than American English. It could probably use a few more English exceptions, too. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24094 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #98735|0 |1 is obsolete| | --- Comment #11 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 99223 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=99223&action=edit Bug 24094: Unit tests Signed-off-by: Myka Kennedy Stephens <mkstephens@lancasterseminary.edu> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24094 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #98736|0 |1 is obsolete| | --- Comment #12 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 99224 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=99224&action=edit Bug 24094: Strip trailing spaces and punctuation from authority headings Both when searching for and creating new authorities we need to remove punctuation that exists in the bibliographic record but does not belong in the authority record. For example, a series with a volume contains a semicolon in the bib record, however, this should not be passed to the authority as the volume is not included in the authority record. To test: 1 - Set AutoCreateAuthorities to 'generate' 2 - Set BiblioAddsAuthorities to 'true' 3 - Set CatalogModuleRelink to 'Do' 4 - Find or create a record with: a 100 field with a subfield e preceded by a comma: 100 $aBoring, M. Eugene M ,$e author an 830 field with a volume preceded by a semicolon: 650$aLord of the rings ;$v 3. 5 - Save the records and check the links 6 - Note punctuation is passed through 7 - Save again, auth records are created again 8 - Apply patch 9 - Save again, new auth records are created again 10 - Check the records, punctuation has been removed 11 - Save again, no more records created. Signed-off-by: Myka Kennedy Stephens <mkstephens@lancasterseminary.edu> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24094 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |martin.renvoize@ptfs-europe | |.com Status|Signed Off |Passed QA --- Comment #13 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Works well, QA scripts are happy and it's got unit tests. Passing QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24094 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |20.05.00 released in| | Status|Passed QA |Pushed to master -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24094 --- Comment #14 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Nice work everyone! Pushed to master for 20.05 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24094 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=24269 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24094 Joy Nelson <joy@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |joy@bywatersolutions.com Status|Pushed to master |Pushed to stable Version(s)|20.05.00 |20.05.00, 19.11.04 released in| | --- Comment #15 from Joy Nelson <joy@bywatersolutions.com> --- Pushed to 19.11.x branch for 19.11.04 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24094 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to stable |Pushed to oldstable CC| |lucas@bywatersolutions.com Version(s)|20.05.00, 19.11.04 |20.05.00, 19.11.04, released in| |19.05.09 --- Comment #16 from Lucas Gass <lucas@bywatersolutions.com> --- backported to 19.05.x for 19.05.09 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24094 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=33100 -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org