[Bug 24827] New: MARC preview fails when staged record contains items with UTF-8 characters
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24827 Bug ID: 24827 Summary: MARC preview fails when staged record contains items with UTF-8 characters Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: MARC Bibliographic record staging/import Assignee: koha-bugs@lists.koha-community.org Reporter: a.roussos@dataly.gr QA Contact: testopia@bugs.koha-community.org The MARC preview of staged records fails when a staged record contains items with UTF-8 characters (see attached screenshots). That's because in EmbedItemsInImportBiblio() the call to MARC::Record->new_from_xml() does not specify 'UTF-8' as the optional encoding to use. In intranet-error.log you'll get messages such as: showmarc.pl: XSLT_Handler: :63: parser error : PCDATA invalid Char value 27 showmarc.pl: <subfield code="o">889.3 \x1b(SVAU\x1bs</subfield> I suggest we fix this by explicitly specifying 'UTF-8' as the encoding to use whenever a call to new_from_xml() is made across the codebase. For consistency, I also propose to standardise on the encoding name in direct calls to new_from_xml(), and use 'UTF-8' from now on instead of 'utf8' or 'utf-8'. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24827 Andreas Roussos <a.roussos@dataly.gr> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED Assignee|koha-bugs@lists.koha-commun |a.roussos@dataly.gr |ity.org | --- Comment #1 from Andreas Roussos <a.roussos@dataly.gr> --- Created attachment 100313 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=100313&action=edit MARC preview failure -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24827 --- Comment #2 from Andreas Roussos <a.roussos@dataly.gr> --- Created attachment 100314 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=100314&action=edit MARC preview working OK -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24827 --- Comment #3 from Andreas Roussos <a.roussos@dataly.gr> --- Created attachment 100315 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=100315&action=edit test record with embedded items containing UTF-8 data -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24827 Andreas Roussos <a.roussos@dataly.gr> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Needs Signoff Patch complexity|--- |Small patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24827 --- Comment #4 from Andreas Roussos <a.roussos@dataly.gr> --- Created attachment 100320 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=100320&action=edit Bug 24827: Specify UTF-8 as encoding in new_from_xml() The MARC preview of staged records fails when a staged record contains items with UTF-8 characters (see attached screenshots). That's because in EmbedItemsInImportBiblio() the call to MARC::Record->new_from_xml() does not explicitly specify 'UTF-8' as the optional encoding to use. This patch fixes that, and also standardises the encoding name used in direct calls to new_from_xml() to 'UTF-8' instead of 'utf8' or 'utf-8'. Test plan (in a MARC21 instance): 0) Stage for import the test record attached to this bug report (make sure that 'Check for embedded item record data?' is set to 'Yes'), then go to Home > Tools > Manage staged MARC records and click on the filename you've just staged. 1) In the 'Citation' column, click on the hyperlink to launch the MARC preview -- notice how it fails to produce any output apart from the modal heading. 2) Apply the patch, and restart Plack if necessary. 3) Re-launch the MARC preview: this time the output should include the whole record, including the embedded item data in field 952. 4) Run the unit tests to ensure nothing was broken inadvertently: $ prove t $ prove xt -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24827 Owen Leonard <oleonard@myacpl.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24827 Owen Leonard <oleonard@myacpl.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #100320|0 |1 is obsolete| | --- Comment #5 from Owen Leonard <oleonard@myacpl.org> --- Created attachment 100410 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=100410&action=edit Bug 24827: Specify UTF-8 as encoding in new_from_xml() The MARC preview of staged records fails when a staged record contains items with UTF-8 characters (see attached screenshots). That's because in EmbedItemsInImportBiblio() the call to MARC::Record->new_from_xml() does not explicitly specify 'UTF-8' as the optional encoding to use. This patch fixes that, and also standardises the encoding name used in direct calls to new_from_xml() to 'UTF-8' instead of 'utf8' or 'utf-8'. Test plan (in a MARC21 instance): 0) Stage for import the test record attached to this bug report (make sure that 'Check for embedded item record data?' is set to 'Yes'), then go to Home > Tools > Manage staged MARC records and click on the filename you've just staged. 1) In the 'Citation' column, click on the hyperlink to launch the MARC preview -- notice how it fails to produce any output apart from the modal heading. 2) Apply the patch, and restart Plack if necessary. 3) Re-launch the MARC preview: this time the output should include the whole record, including the embedded item data in field 952. 4) Run the unit tests to ensure nothing was broken inadvertently: $ prove t $ prove xt Signed-off-by: Owen Leonard <oleonard@myacpl.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24827 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@bugs.koha-c | |ommunity.org Status|Signed Off |Failed QA --- Comment #6 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Andreas, I agree with the changes. But could split the patch into 2 parts: 1 for the bugfix, 1 for the enhancement. That way the backport will not cause conflict (only the bugfix could be backported). Switch back the status to "Signed Off" when done. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24827 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |6520 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6520 [Bug 6520] Items don't display when viewing staged records -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24827 Andreas Roussos <a.roussos@dataly.gr> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #100410|0 |1 is obsolete| | --- Comment #7 from Andreas Roussos <a.roussos@dataly.gr> --- Created attachment 100694 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=100694&action=edit Bug 24827: Standardise on 'UTF-8' as the encoding name This patch standardises the encoding name used in direct calls to new_from_xml() to 'UTF-8' instead of 'utf8' or 'utf-8'. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24827 --- Comment #8 from Andreas Roussos <a.roussos@dataly.gr> --- Created attachment 100695 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=100695&action=edit Bug 24827: Specify UTF-8 as encoding in new_from_xml() The MARC preview of staged records fails when a staged record contains items with UTF-8 characters (see attached screenshots). That's because in EmbedItemsInImportBiblio() the call to MARC::Record->new_from_xml() does not explicitly specify 'UTF-8' as the optional encoding to use. This patch fixes that. Test plan (in a MARC21 instance): 0) Stage for import the test record attached to this bug report (make sure that 'Check for embedded item record data?' is set to 'Yes'), then go to Home > Tools > Manage staged MARC records and click on the filename you've just staged. 1) In the 'Citation' column, click on the hyperlink to launch the MARC preview -- notice how it fails to produce any output apart from the modal heading. 2) Apply the patch, and restart Plack if necessary. 3) Re-launch the MARC preview: this time the output should include the whole record, including the embedded item data in field 952. 4) Run the unit tests to ensure nothing was broken inadvertently: $ prove t $ prove xt -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24827 Andreas Roussos <a.roussos@dataly.gr> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Signed Off --- Comment #9 from Andreas Roussos <a.roussos@dataly.gr> --- Thanks for your feedback, Jonathan. I've split my initial patch in two, hopefully I got this right ;-) Setting Status to "Signed Off". -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24827 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24827 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #100694|0 |1 is obsolete| | Attachment #100695|0 |1 is obsolete| | --- Comment #10 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 100737 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=100737&action=edit Bug 24827: Standardise on 'UTF-8' as the encoding name This patch standardises the encoding name used in direct calls to new_from_xml() to 'UTF-8' instead of 'utf8' or 'utf-8'. Signed-off-by: Owen Leonard <oleonard@myacpl.org> Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24827 --- Comment #11 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 100738 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=100738&action=edit Bug 24827: Specify UTF-8 as encoding in new_from_xml() The MARC preview of staged records fails when a staged record contains items with UTF-8 characters (see attached screenshots). That's because in EmbedItemsInImportBiblio() the call to MARC::Record->new_from_xml() does not explicitly specify 'UTF-8' as the optional encoding to use. This patch fixes that. Test plan (in a MARC21 instance): 0) Stage for import the test record attached to this bug report (make sure that 'Check for embedded item record data?' is set to 'Yes'), then go to Home > Tools > Manage staged MARC records and click on the filename you've just staged. 1) In the 'Citation' column, click on the hyperlink to launch the MARC preview -- notice how it fails to produce any output apart from the modal heading. 2) Apply the patch, and restart Plack if necessary. 3) Re-launch the MARC preview: this time the output should include the whole record, including the embedded item data in field 952. 4) Run the unit tests to ensure nothing was broken inadvertently: $ prove t $ prove xt Signed-off-by: Owen Leonard <oleonard@myacpl.org> Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24827 --- Comment #12 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 100739 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=100739&action=edit Bug 24827: Add a test Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24827 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |20.05.00 released in| | Status|Passed QA |Pushed to master -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24827 --- Comment #13 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Nice work everyone! Pushed to master for 20.05 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24827 Joy Nelson <joy@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to master |Pushed to stable CC| |joy@bywatersolutions.com Version(s)|20.05.00 |20.05.00, 19.11.05 released in| | --- Comment #14 from Joy Nelson <joy@bywatersolutions.com> --- backported to 19.11.x branch for 19.11.05 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24827 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lucas@bywatersolutions.com --- Comment #15 from Lucas Gass <lucas@bywatersolutions.com> --- patchset doesnt apply clean to 19.05.x, please rebase if needed -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org