[Bug 16370] New: Set the record encoding to UTF8 when TransformHtmlToMarc
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16370 Bug ID: 16370 Summary: Set the record encoding to UTF8 when TransformHtmlToMarc Change sponsored?: --- Product: Koha Version: 3.22 Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: MARC Bibliographic data support Assignee: gmcharlt@gmail.com Reporter: cdufour-keyword-koha.b708e6@idiap.ch QA Contact: testopia@bugs.koha-community.org Created attachment 50819 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=50819&action=edit TransformHtmlToMarc-UTF8.patch Hello, Using Koha 3.22.05 (Debian 8 packaging), I noticed that creating a new authority record in the web interface would lead to character encoding inconsistencies when the leader is not included in the MARC editor (which I felt is not necessary for authority records): in the database, the "auth_header.marc" (blob) would still have a "MARC-8" leader, despite the content being correctly UTF-8 encoded. I tracked the issue down to the new MARC record not being initialized as UTF-8 encoded when being created from the web interface (=> TransformHtmlToMarc). The one-liner attached patch solved that issue. Best, Cédric -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16370 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@bugs.koha-c | |ommunity.org --- Comment #1 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Hi Cédric, Please have a look at the following wiki patch to know how to correctly format a patch: https://wiki.koha-community.org/wiki/SubmitingAPatch It would be also good to provide a test plan to reproduce the issue. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16370 Liz Rea <liz@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |liz@catalyst.net.nz --- Comment #2 from Liz Rea <liz@catalyst.net.nz> --- I can tell you how to reproduce this, it happens in the standard MARC editor as well as the authority editor. Create a framework that doesn't include the 000 in the editor (set to ignore) alternatively, import the attached framework. create or import a record that has UTF8 characters in it into that framework. I used āāāāāāā for the title, because that's easy to spot disappearing. Edit the record, and modify any field. You'll notice that the title is no longer displayed after you save it. If you look in the database in biblioitems for this biblio, you'll spot that the MARC has the title saved, but the MARCXML does not. As Cédric says, it seems to be because the leader is set to MARC-8 in the absence of the leader, instead of UTF8. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16370 --- Comment #3 from Liz Rea <liz@catalyst.net.nz> --- Created attachment 53254 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=53254&action=edit Framework for testing, no 000 visible -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16370 Liz Rea <liz@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff Patch complexity|--- |Small patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16370 --- Comment #4 from Liz Rea <liz@catalyst.net.nz> --- Created attachment 53255 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=53255&action=edit Bug 16370 - Frameworks without 000 visible won't save UTF8 characters Create a framework that doesn't include the 000 in the editor (set to ignore) alternatively, import the attached framework. Create or import a record that has UTF8 characters in it into that framework. I used āāāāāāā for the title, because that's easy to spot disappearing. Edit the record, and modify any field. You'll notice that the title is no longer displayed after you save it. Apply the patch, redo above, fields with UTF8 Characters will be saved correctly Note (from Liz): I have no idea if this is the right solution, but it does fix the problem, so I'm formatting this as required for testing/inclusion on behalf of Cédric. If you look in the database in biblioitems for this biblio, you'll spot that the MARC has the title saved, but the MARCXML does not. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16370 Liz Rea <liz@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #53255|0 |1 is obsolete| | --- Comment #5 from Liz Rea <liz@catalyst.net.nz> --- Created attachment 53256 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=53256&action=edit Bug 16370 - Frameworks without 000 visible won't save UTF8 characters Create a framework that doesn't include the 000 in the editor (set to ignore) alternatively, import the attached framework. Create or import a record that has UTF8 characters in it into that framework. I used āāāāāāā for the title, because that's easy to spot disappearing. Edit the record, and modify any field. You'll notice that the title is no longer displayed after you save it, and no longer appears in the editor. If you look in the database in biblioitems for this biblio, you'll spot that the MARC has the title saved, but the MARCXML does not. Apply the patch, redo above, fields with UTF8 Characters will be saved correctly Note (from Liz): I have no idea if this is the right solution, but it does fix the problem, so I'm formatting this as required for testing/inclusion on behalf of Cédric. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16370 M. Tompsett <mtompset@hotmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #50819|0 |1 is obsolete| | --- Comment #6 from M. Tompsett <mtompset@hotmail.com> --- Comment on attachment 50819 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=50819 TransformHtmlToMarc-UTF8.patch Liz Rea's patch is identical and git-friendlier for testing. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16370 --- Comment #7 from Liz Rea <liz@catalyst.net.nz> --- I even kept the attribution! :) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16370 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |m.de.rooy@rijksmuseum.nl -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16370 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |In Discussion --- Comment #8 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> ---
Create a framework that doesn't include the 000 in the editor (set to ignore) alternatively, import the attached framework.
Hmm. Isn't this kind of a strange use case? I have doubts on correcting the problem in TransformHtmlToMarc. Currently, the marc21 leader plugin forces the leader position to Unicode. But what if you edit the leader manually? (Just as strange as ignoring the leader.) And what are the implications for UNIMARC? Note that the encoding for that is not in the leader but in field 100. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16370 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Discussion |Failed QA -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org