[Bug 42948] New: Diacritics corrupted when saving authority records (UTF-8 mangled into unrelated Unicode characters, e.g. é→ř, ú/ñ→ž)
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42948 Bug ID: 42948 Summary: Diacritics corrupted when saving authority records (UTF-8 mangled into unrelated Unicode characters, e.g. é→ř, ú/ñ→ž) Initiative type: --- Sponsorship --- status: Product: Koha Version: 26.05 Hardware: PC OS: Linux Status: NEW Severity: major Priority: P5 - low Component: MARC Authority data support Assignee: koha-bugs@lists.koha-community.org Reporter: 328807@unizar.es QA Contact: testopia@bugs.koha-community.org Target Milestone: --- Created attachment 201329 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201329&action=edit Attachments included: 1 Screenshot of authority before save (correct name) 2 Screenshot of authority after save (corrupted name) 3 Screenshot showing the same corruption when typed When creating or saving an authority record containing accented characters (e.g. Spanish names with á, é, í, ó, ú, ñ), the diacritics are corrupted into unrelated Unicode characters upon save. The corruption pattern is not simple mojibake (e.g. "é") but rather diacritics merging with adjacent letters into different Unicode characters, including letters from other alphabets (e.g. ř, ž, ø, æ). Example: Correct name: "Pérez Zúñiga, Juan" After saving the authority in Koha: "Přez Žiga, Juan" Correct name: "García-Posada, Miguel" After saving: "Garcá-Posada, Miguel" (character dropped entirely) Steps to reproduce: Go to Authorities module. Create a new authority manually (New authority), entering a personal name with accented characters directly in field 100 $a (e.g. "Hierro, José" or "Pérez Zúñiga, Juan"). Save the record. Reopen the saved authority record. Observe that the accented characters have been replaced/corrupted. This reproduces even when the authority is created entirely manually, without using Z39.50/SRU import, ruling out any Z39.50 target encoding issue. Troubleshooting already performed (ruling out other causes): Confirmed the issue is NOT related to the Z39.50/BNE server encoding: tested with UTF-8, MARC-8, ISO-5426, ISO-8859-1 and ISO_6937 encodings on the Z39.50 target — all produce different (but still wrong) corruption patterns, and the corruption also occurs when manually typing the authority without using Z39.50 at all. Confirmed the Leader position 09 (Character coding scheme) of the bibliographic record is correctly set to "a" (UCS/Unicode), auto-filled. Confirmed the same corruption occurs when importing the same correct UTF-8 data from a different Z39.50 target (Library of Congress Names), which displays the name correctly in the search results list, but the data is still corrupted once saved as a Koha authority — pointing to the save process itself, not the data source. Confirmed database-level encoding is correctly configured throughout: Database: utf8mb4 / utf8mb4_uca1400_ai_ci Table auth_header: column marcxml is longtext with collation utf8mb4_unicode_ci MySQL/MariaDB session variables (character_set_client, character_set_connection, character_set_database, character_set_results, character_set_server) all set to utf8mb4 collation_connection, collation_database, collation_server all utf8mb4_uca1400_ai_ci Given the above, the corruption appears to originate in the Koha application layer (Perl/Plack) during the authority save process, rather than in any Z39.50 target configuration or in the database encoding configuration. Environment: Koha version: 26.05.00 Database: MariaDB (utf8mb4, collation utf8mb4_uca1400_ai_ci) Authority framework: MARC21 PERSO_NAME Attachments included: • Screenshot of authority before save (correct name) • Screenshot of authority after save (corrupted name) • Screenshot showing the same corruption when typed manually (no Z39.50 involved) -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42948 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au --- Comment #1 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Hi Sofia, this sounds really odd. You might want to also ask on Mattermost Chat - I believe it's more likely a setup issue than an issue with Koha, as we should have had a lot more reports otherwise. And it might be a little easier to help you there by checking some things (https://chat.koha-community.org Some questions to maybe help narrow it down (sorry if I missed it in your text): Your example is for authorities - so the problem doesn't exist with bibliographic records, correct? Is this a new installation or an older/updated one? If older - did it break after a certain update? You are using MARC21 - correct? -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42948 --- Comment #2 from Sofia <328807@unizar.es> --- Q-Your example is for authorities - so the problem doesn't exist with bibliographic records, correct? A- That’s correct. Only for authorities; it is always detected when there’s tipical spanish stuff, like diacritics or accents (like jamón) or letter “ñ”, for example. The same if written directly or from Z39.50, it happens just after clicking “Save”. Q-Is this a new installation or an older/updated one? A-New installation Q-If older - did it break after a certain update? A-N/A. Just from the beginning when I have started creating authorities. Q- You are using MARC21 – correct? A- Correct -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42948 --- Comment #3 from David Cook <dcook@prosentient.com.au> --- Thanks for the CC Katrin. I love my encoding problems. I'll take a look. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42948 --- Comment #4 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to David Cook from comment #3)
Thanks for the CC Katrin. I love my encoding problems. I'll take a look.
I was hoping you'd say that :) Thanks David! -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42948 --- Comment #5 from David Cook <dcook@prosentient.com.au> --- (In reply to Sofia from comment #0)
Koha version: 26.05.00 Database: MariaDB (utf8mb4, collation utf8mb4_uca1400_ai_ci) Authority framework: MARC21 PERSO_NAME
Hmm I can't reproduce on main with English... next I'll try 26.05.00 with English and if it still works I'll try with Spanish... -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42948 --- Comment #6 from David Cook <dcook@prosentient.com.au> --- (In reply to David Cook from comment #5)
(In reply to Sofia from comment #0)
Koha version: 26.05.00 Database: MariaDB (utf8mb4, collation utf8mb4_uca1400_ai_ci) Authority framework: MARC21 PERSO_NAME
Hmm I can't reproduce on main with English... next I'll try 26.05.00 with English and if it still works I'll try with Spanish...
Can't reproduce on 26.05.00 either in English in KTD... it shows up as "Pérez Zúñiga, Juan" regardless. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42948 --- Comment #7 from David Cook <dcook@prosentient.com.au> --- (In reply to David Cook from comment #6)
(In reply to David Cook from comment #5)
(In reply to Sofia from comment #0)
Koha version: 26.05.00 Database: MariaDB (utf8mb4, collation utf8mb4_uca1400_ai_ci) Authority framework: MARC21 PERSO_NAME
Hmm I can't reproduce on main with English... next I'll try 26.05.00 with English and if it still works I'll try with Spanish...
Can't reproduce on 26.05.00 either in English in KTD... it shows up as "Pérez Zúñiga, Juan" regardless.
I do wonder a little bit about that collation but that shouldn't be an issue since it's just the collation and not the storage... I'll install Spanish and give that a go. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42948 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Diacritics corrupted when |Authority records default |saving authority records |to MARC-8 instead of UTF-8 |(UTF-8 mangled into |if missing MARC21 leader |unrelated Unicode | |characters, e.g. é→ř, | |ú/ñ→ž) | Status|NEW |CONFIRMED --- Comment #8 from David Cook <dcook@prosentient.com.au> --- So if I change the Leader to use MARC-8 encoding instead of UTF-8 I get "Přez Zį͠ga, Juan" which is different to what you said on Bugzilla but is the same to what I see in your screenshots with the Z39.50. And looking at your screenshots again... you've got a misconfigured framework for your "Authority type". You're missing the "000 - LEADER". And if there is no "000 - LEADER", it looks like it doesn't default to UTF-8 and instead defaults to MARC-8, which then leads to your encoding issue. -- I had wondered a bit at the start if this was more of a support issue than a Koha bug, but I thought it worth investigating. I should've double-checked the screenshots earlier... I think... we can probably still classify this one as a bug, because all MARC21 records should have a leader. Maybe we do need to make certain fields unhideable (e.g. 000/Leader, 005, 008) as hiding these fields leads to bad records. I'm going to rename this one from 'Diacritics corrupted when saving authority records (UTF-8 mangled into unrelated Unicode characters, e.g. é→ř, ú/ñ→ž)' to 'Authority records default to MARC-8 instead of UTF-8 if missing MARC21 leader' -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42948 --- Comment #9 from David Cook <dcook@prosentient.com.au> --- I suspect this is also an issue with all versions of Koha and not just 26.05 although I haven't confirmed yet. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42948 --- Comment #10 from David Cook <dcook@prosentient.com.au> --- Probably also an issue with Bib records although I haven't checked that either yet -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42948 Phil Ringnalda <phil@chetcolibrary.org> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=43048 -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42948 --- Comment #11 from David Cook <dcook@prosentient.com.au> --- (In reply to David Cook from comment #10)
Probably also an issue with Bib records although I haven't checked that either yet
Funny enough we bumped into this doing a migration last Thursday heh -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42948 Esther Melander <esther.melander@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |esther.melander@bywatersolu | |tions.com -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org