http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9114 --- Comment #1 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 22740 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=22740&action=edit Bug 9114: import/export routines should use UTF-8 Currently both the import_export_framework.pl script and the associated libs output data with Perl's default encoding, ISO-8859. This patch properly sets the binmode to UTF-8 for the CGI script, and makes the C4::ImportExportFramework routines apply the ":encoding(UTF-8)" layer to file descriptors. To test: Export step test - Use some non-ASCII character(s) in some field description in a chosen framework. - Export the framework at Administration > MARC frameworks - Run this to check the file is ISO-8859 encoded: $ file export_XXX.csv export_XXX.csv: ISO-8859 text, with very long lines (Note: try SQL and other output formats too. But not ODS) - Apply the patch - Export the framework again (change the name), and test encoding: $ file export_XXX_2.csv export_XXX_2.csv: UTF-8 Unicode text Import step test I assume you have two files, export_XXX.csv (ISO-8859 encoded) and export_XXX_2.csv (XXX will depend on your framework's code) - Reset your testing branch to master - Import export_XXX.csv - The string with non-ASCII chars is truncated at the first non-ASCII char's position (Note: this is the current behaviour). - Import export_XXX_2.csv - The non-ASCII chars are broken, the logs show errors on non-UNICODE chars. (Note: even thou UTF-8 is the expected encoding it is treated as ISO-8859). - Apply the patch - Import the good (UTF-8 as expected) file and check everything worked as expected. Regards To+ Sponsored-by: Universidad Nacional de Cordoba -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9114 mathieu saby <mathieu.saby@univ-rennes2.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mathieu.saby@univ-rennes2.f | |r --- Comment #2 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- Hello WITHOUT your patch, when I run $ file ~/Téléchargements/export_.csv I get: /home/msaby/Téléchargements/export_.csv: UTF-8 Unicode text I also put some chinese characters in the name of a field, and when I import the saved framework, I can see those characters in the new framework in Koha. So, can you confirm the issue, and your test plan? Mathieu -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9114 --- Comment #4 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to M. de Rooy from comment #3)

Tomas: Thank you for your patch. I have a comment however :) You use binmode on STDOUT in the module, but you print to a file handle. So that should not be enough. I would suggest to replace the binmode by adding the encoding to the output file handles. Note that I also had a wide character warning in my log on admin/import_export_framework. You probably need to touch code there as well? I tested with a Chinese character; that went well (export and re-import). Since you actually leave it to Perl to decide the encoding, Perl made the right choice here. But with some Latin characters only, export and re-import made them disappear..

Failed QA

Some confusion: The binmode is not in the module, but printing to the file handle is. So please correct. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9114 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9114 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Version|unspecified |master -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9114 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |kyle@bywatersolutions.com Severity|enhancement |minor -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9114 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #24450|0 |1 is obsolete| | --- Comment #7 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 24514 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=24514&action=edit Bug 9114: import/export routines should use UTF-8 Currently both the import_export_framework.pl script and the associated libs output data with Perl's default encoding, ISO-8859. This patch properly sets the binmode to UTF-8 for the CGI script, and makes the C4::ImportExportFramework routines apply the ":encoding(UTF-8)" layer to file descriptors. To test: Export step test - Use some non-ASCII character(s) in some field description in a chosen framework. - Export the framework at Administration > MARC frameworks - Run this to check the file is ISO-8859 encoded: $ file export_XXX.csv export_XXX.csv: ISO-8859 text, with very long lines (Note: try SQL and other output formats too. But not ODS) - Apply the patch - Export the framework again (change the name), and test encoding: $ file export_XXX_2.csv export_XXX_2.csv: UTF-8 Unicode text Import step test I assume you have two files, export_XXX.csv (ISO-8859 encoded) and export_XXX_2.csv (XXX will depend on your framework's code) - Reset your testing branch to master - Import export_XXX.csv - The string with non-ASCII chars is truncated at the first non-ASCII char's position (Note: this is the current behaviour). - Import export_XXX_2.csv - The non-ASCII chars are broken, the logs show errors on non-UNICODE chars. (Note: even thou UTF-8 is the expected encoding it is treated as ISO-8859). - Apply the patch - Import the good (UTF-8 as expected) file and check everything worked as expected. Regards To+ Sponsored-by: Universidad Nacional de Cordoba Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Tested export and re-import with CSV containing Latin chars and containing Chinese chars. Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.

Marking failed QA for the regressions.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9114 --- Comment #9 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to Galen Charlton from comment #8) thx for scrutiny++ -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9114 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #24514|0 |1 is obsolete| | --- Comment #11 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 24630 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=24630&action=edit Bug 9114: Make frameworks import/export routines correctly use UTF-8 Currently both the import_export_framework.pl script outputs data with Perl's default encoding, ISO-8859. This patch properly sets the binmode to UTF-8 when exporting SQL and CSV files using the PerlIO layer (":encoding(UTF-8)") for STDOUT. To test: Export step test - Use some ASCII character(s) with DIACRITICS in some field description in a chosen framework. - Export the framework at Administration > MARC frameworks - Run this to check the file is ISO-8859 encoded: $ file export_XXX.csv export_XXX.csv: ISO-8859 text, with very long lines (Note: try SQL and other output formats too. But not ODS) - Apply the patch - Export the framework again (change the name), and test encoding: $ file export_XXX_2.csv export_XXX_2.csv: UTF-8 Unicode text Import step test I assume you have two files, export_XXX.csv (ISO-8859 encoded) and export_XXX_2.csv (XXX will depend on your framework's code) - Reset your testing branch to master - Import export_XXX.csv - The string with non-ASCII chars is truncated at the first non-ASCII char's position (Note: this is the current behaviour). - Import export_XXX_2.csv - The non-ASCII chars are broken, the logs show errors on non-UNICODE chars. (Note: even thou UTF-8 is the expected encoding it is treated as ISO-8859). - Apply the patch - Import the good (UTF-8 as expected) file and check everything worked as expected. No double encoding should occur with either combination of formats. Regards To+ Sponsored-by: Universidad Nacional de Cordoba -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9114 Magnus Enger <magnus@enger.priv.no> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9114 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #26285|0 |1 is obsolete| | --- Comment #15 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 26320 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=26320&action=edit Bug 9114: Make frameworks import/export routines correctly use UTF-8 Currently both the import_export_framework.pl script outputs data with Perl's default encoding, ISO-8859. This patch properly sets the binmode to UTF-8 when exporting SQL and CSV files using the PerlIO layer (":encoding(UTF-8)") for STDOUT. To test: Export step test - Use some ASCII character(s) with DIACRITICS in some field description in a chosen framework. - Export the framework at Administration > MARC frameworks - Run this to check the file is ISO-8859 encoded: $ file export_XXX.csv export_XXX.csv: ISO-8859 text, with very long lines (Note: try SQL and other output formats too. But not ODS) - Apply the patch - Export the framework again (change the name), and test encoding: $ file export_XXX_2.csv export_XXX_2.csv: UTF-8 Unicode text Import step test I assume you have two files, export_XXX.csv (ISO-8859 encoded) and export_XXX_2.csv (XXX will depend on your framework's code) - Reset your testing branch to master - Import export_XXX.csv - The string with non-ASCII chars is truncated at the first non-ASCII char's position (Note: this is the current behaviour). - Import export_XXX_2.csv - The non-ASCII chars are broken, the logs show errors on non-UNICODE chars. (Note: even thou UTF-8 is the expected encoding it is treated as ISO-8859). - Apply the patch - Import the good (UTF-8 as expected) file and check everything worked as expected. No double encoding should occur with either combination of formats. Regards To+ Sponsored-by: Universidad Nacional de Cordoba Signed-off-by: Magnus Enger <digitalutvikling@gmail.com> I put some Norwegian and accented letters in a fremawork to test. Before the patch, the exported CSV came out as ISO-8859, after the patch it came out as UTF-8. ODS and XML (viewed in LibreOffice) both looked good, before and after the patch. Importing the ISO-8859 CSV cut off the strings at the first non-ASCII char. Importing the UTF-8 CSV worked as epected. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9114 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #26320|0 |1 is obsolete| | --- Comment #16 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 26350 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=26350&action=edit [PASSED QA] Bug 9114: Make frameworks import/export routines correctly use UTF-8 Currently both the import_export_framework.pl script outputs data with Perl's default encoding, ISO-8859. This patch properly sets the binmode to UTF-8 when exporting SQL and CSV files using the PerlIO layer (":encoding(UTF-8)") for STDOUT. To test: Export step test - Use some ASCII character(s) with DIACRITICS in some field description in a chosen framework. - Export the framework at Administration > MARC frameworks - Run this to check the file is ISO-8859 encoded: $ file export_XXX.csv export_XXX.csv: ISO-8859 text, with very long lines (Note: try SQL and other output formats too. But not ODS) - Apply the patch - Export the framework again (change the name), and test encoding: $ file export_XXX_2.csv export_XXX_2.csv: UTF-8 Unicode text Import step test I assume you have two files, export_XXX.csv (ISO-8859 encoded) and export_XXX_2.csv (XXX will depend on your framework's code) - Reset your testing branch to master - Import export_XXX.csv - The string with non-ASCII chars is truncated at the first non-ASCII char's position (Note: this is the current behaviour). - Import export_XXX_2.csv - The non-ASCII chars are broken, the logs show errors on non-UNICODE chars. (Note: even thou UTF-8 is the expected encoding it is treated as ISO-8859). - Apply the patch - Import the good (UTF-8 as expected) file and check everything worked as expected. No double encoding should occur with either combination of formats. Regards To+ Sponsored-by: Universidad Nacional de Cordoba Signed-off-by: Magnus Enger <digitalutvikling@gmail.com> I put some Norwegian and accented letters in a fremawork to test. Before the patch, the exported CSV came out as ISO-8859, after the patch it came out as UTF-8. ODS and XML (viewed in LibreOffice) both looked good, before and after the patch. Importing the ISO-8859 CSV cut off the strings at the first non-ASCII char. Importing the UTF-8 CSV worked as epected. Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de> Works as expected, passes tests and QA script. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9114 Fridolin SOMERS <fridolyn.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to Master |Pushed to Stable CC| |fridolyn.somers@biblibre.co | |m --- Comment #18 from Fridolin SOMERS <fridolyn.somers@biblibre.com> --- Pushed to 3.14.x, will be in 3.14.08 -- You are receiving this mail because: You are watching all bug changes.