[Bug 32012] New: runreport.pl should use binmode UTF-8
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32012 Bug ID: 32012 Summary: runreport.pl should use binmode UTF-8 Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: major Priority: P5 - low Component: Command-line Utilities Assignee: koha-bugs@lists.koha-community.org Reporter: fridolin.somers@biblibre.com QA Contact: testopia@bugs.koha-community.org CC: robin@catalyst.net.nz Like Bug 26832 added binmode UTF-8 to script misc/search_tools/export_elasticsearch_mappings.pl, this should be added to misc/cronjobs/runreport.pl. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32012 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |22323 Status|NEW |ASSIGNED Assignee|koha-bugs@lists.koha-commun |fridolin.somers@biblibre.co |ity.org |m --- Comment #1 from Fridolin Somers <fridolin.somers@biblibre.com> --- Strange this was not done with Bug 22323 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22323 [Bug 22323] Cronjob runreport.pl has a CSV encoding issue -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32012 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=26832 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32012 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Needs Signoff Patch complexity|--- |Trivial patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32012 --- Comment #2 from Fridolin Somers <fridolin.somers@biblibre.com> --- Created attachment 142701 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=142701&action=edit Bug 32012: runreport.pl should use binmode UTF-8 Like Bug 26832 added binmode UTF-8 to script misc/search_tools/export_elasticsearch_mappings.pl, this should be added to misc/cronjobs/runreport.pl. Test plan : 1) Do not apply patch 2) Create a SQL report with : S SELECT 'accentué',barcode FROM items limit 3 3) Note the id of this report, for example 1 4) Run : misc/cronjobs/runreport.pl 1 --format csv | tee /tmp/without.csv => You see output with unknown char instead of é : accentu<?>,1 accentu<?>,2 accentu<?>,3 5) Run : file --mime-type /tmp/without.csv => You see : /tmp/without.csv: iso-8859-1 6) Apply patch 7) Run : misc/cronjobs/runreport.pl 1 --format csv | tee /tmp/with.csv => You see correct output : accentuÃ,1 accentuÃ,2 accentuÃ,3 8) Run : file --mime-type /tmp/without.csv => You see : /tmp/without.csv: utf-8 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32012 --- Comment #3 from Fridolin Somers <fridolin.somers@biblibre.com> --- Comment on attachment 142701 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=142701 Bug 32012: runreport.pl should use binmode UTF-8
From ef19187a6588da31d0706e73a6d8f45754b30c30 Mon Sep 17 00:00:00 2001 From: Fridolin Somers <fridolin.somers@biblibre.com> Date: Wed, 26 Oct 2022 23:15:35 -1000 Subject: [PATCH] Bug 32012: runreport.pl should use binmode UTF-8 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit
Like Bug 26832 added binmode UTF-8 to script misc/search_tools/export_elasticsearch_mappings.pl, this should be added to misc/cronjobs/runreport.pl.
Test plan : 1) Do not apply patch 2) Create a SQL report with : S SELECT 'accentué',barcode FROM items limit 3 3) Note the id of this report, for example 1 4) Run : misc/cronjobs/runreport.pl 1 --format csv | tee /tmp/without.csv => You see output with unknown char instead of é : accentu<?>,1 accentu<?>,2 accentu<?>,3 5) Run : file --mime-type /tmp/without.csv => You see : /tmp/without.csv: iso-8859-1 6) Apply patch 7) Run : misc/cronjobs/runreport.pl 1 --format csv | tee /tmp/with.csv => You see correct output : accentué,1 accentué,2 accentué,3 8) Run : file --mime-type /tmp/without.csv => You see : /tmp/without.csv: utf-8 --- misc/cronjobs/runreport.pl | 2 ++ 1 file changed, 2 insertions(+)
diff --git a/misc/cronjobs/runreport.pl b/misc/cronjobs/runreport.pl index d6fbb7fcbc..8b3c12b3b0 100755 --- a/misc/cronjobs/runreport.pl +++ b/misc/cronjobs/runreport.pl @@ -164,6 +164,8 @@ Reports - Guided Reports
=cut
+binmode STDOUT, ":encoding(UTF-8)"; + # These variables can be set by command line options, # initially set to default values.
-- 2.35.5
-- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32012 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #142701|0 |1 is obsolete| | --- Comment #4 from Fridolin Somers <fridolin.somers@biblibre.com> --- Created attachment 142702 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=142702&action=edit Bug 32012: runreport.pl should use binmode UTF-8 Like Bug 26832 added binmode UTF-8 to script misc/search_tools/export_elasticsearch_mappings.pl, this should be added to misc/cronjobs/runreport.pl. Test plan : 1) Do not apply patch 2) Create a SQL report with : S SELECT 'accentué',barcode FROM items limit 3 3) Note the id of this report, for example 1 4) Run : misc/cronjobs/runreport.pl 1 --format csv | tee /tmp/without.csv => You see output with unknown character instead of é : 5) Run : file --mime-type /tmp/without.csv => You see : /tmp/without.csv: iso-8859-1 6) Apply patch 7) Run : misc/cronjobs/runreport.pl 1 --format csv | tee /tmp/with.csv => You see correct output with é 8) Run : file --mime-type /tmp/without.csv => You see : /tmp/without.csv: utf-8 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32012 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #142702|0 |1 is obsolete| | --- Comment #5 from Fridolin Somers <fridolin.somers@biblibre.com> --- Created attachment 142703 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=142703&action=edit Bug 32012: runreport.pl should use binmode UTF-8 Like Bug 26832 added binmode UTF-8 to script misc/search_tools/export_elasticsearch_mappings.pl, this should be added to misc/cronjobs/runreport.pl. Test plan : 1) Do not apply patch 2) Create a SQL report with : SELECT 'accentué',barcode FROM items limit 3 3) Note the id of this report, for example 1 4) Run : misc/cronjobs/runreport.pl 1 --format csv | tee /tmp/without.csv => You see output with unknown character instead of é : 5) Run : file --mime-type /tmp/without.csv => You see : /tmp/without.csv: iso-8859-1 6) Apply patch 7) Run : misc/cronjobs/runreport.pl 1 --format csv | tee /tmp/with.csv => You see correct output with é 8) Run : file --mime-type /tmp/without.csv => You see : /tmp/without.csv: utf-8 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32012 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32012 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #142703|0 |1 is obsolete| | --- Comment #6 from David Nind <david@davidnind.com> --- Created attachment 142704 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=142704&action=edit Bug 32012: runreport.pl should use binmode UTF-8 Like Bug 26832 added binmode UTF-8 to script misc/search_tools/export_elasticsearch_mappings.pl, this should be added to misc/cronjobs/runreport.pl. Test plan : 1) Do not apply patch 2) Create a SQL report with : SELECT 'accentué',barcode FROM items limit 3 3) Note the id of this report, for example 1 4) Run : misc/cronjobs/runreport.pl 1 --format csv | tee /tmp/without.csv => You see output with unknown character instead of é : 5) Run : file --mime-type /tmp/without.csv => You see : /tmp/without.csv: iso-8859-1 6) Apply patch 7) Run : misc/cronjobs/runreport.pl 1 --format csv | tee /tmp/with.csv => You see correct output with é 8) Run : file --mime-type /tmp/without.csv => You see : /tmp/without.csv: utf-8 Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32012 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david@davidnind.com --- Comment #7 from David Nind <david@davidnind.com> --- Testing notes using koha-testing-docker. To display the mime type I ran the commands: - Step 5: file /tmp/without.csv -i - Step 8: file /tmp/with.csv -i Using file --mime-type didn't display the mime type for me. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32012 --- Comment #8 from Fridolin Somers <fridolin.somers@biblibre.com> --- (In reply to David Nind from comment #7)
Testing notes using koha-testing-docker.
To display the mime type I ran the commands: - Step 5: file /tmp/without.csv -i - Step 8: file /tmp/with.csv -i
Using file --mime-type didn't display the mime type for me.
Ah ok. Thanks a lot for testing -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32012 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #142704|0 |1 is obsolete| | --- Comment #9 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 142986 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=142986&action=edit Bug 32012: runreport.pl should use binmode UTF-8 Like Bug 26832 added binmode UTF-8 to script misc/search_tools/export_elasticsearch_mappings.pl, this should be added to misc/cronjobs/runreport.pl. Test plan : 1) Do not apply patch 2) Create a SQL report with : SELECT 'accentué',barcode FROM items limit 3 3) Note the id of this report, for example 1 4) Run : misc/cronjobs/runreport.pl 1 --format csv | tee /tmp/without.csv => You see output with unknown character instead of é : 5) Run : file --mime-type /tmp/without.csv => You see : /tmp/without.csv: iso-8859-1 6) Apply patch 7) Run : misc/cronjobs/runreport.pl 1 --format csv | tee /tmp/with.csv => You see correct output with é 8) Run : file --mime-type /tmp/without.csv => You see : /tmp/without.csv: utf-8 Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32012 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |martin.renvoize@ptfs-europe | |.com Status|Signed Off |Passed QA QA Contact|testopia@bugs.koha-communit |martin.renvoize@ptfs-europe |y.org |.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32012 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |22.11.00 released in| | Status|Passed QA |Pushed to master -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32012 --- Comment #10 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Pushed to master for 22.11. Nice work everyone, thanks! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32012 --- Comment #11 from Fridolin Somers <fridolin.somers@biblibre.com> --- This is great :D thanks everyone. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32012 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to master |Pushed to stable Version(s)|22.11.00 |22.11.00, 22.05.08 released in| | CC| |lucas@bywatersolutions.com --- Comment #12 from Lucas Gass <lucas@bywatersolutions.com> --- Backported to 22.05.x for upcoming 22.05.08 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32012 Arthur Suzuki <arthur.suzuki@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |arthur.suzuki@biblibre.com Version(s)|22.11.00, 22.05.08 |22.11.00, 22.05.08, released in| |21.11.15 Status|Pushed to stable |Pushed to oldstable --- Comment #13 from Arthur Suzuki <arthur.suzuki@biblibre.com> --- applied to 21.11 for 21.11.15 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32012 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=33050 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32012 --- Comment #14 from Fridolin Somers <fridolin.somers@biblibre.com> --- Ah strange, I don't see this commit in 21.11.x : https://git.koha-community.org/Koha-community/Koha/commits/branch/21.11.x/mi... -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org