[Bug 39023] New: Add utf8 output to CSV output of overdue_notices.pl
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39023 Bug ID: 39023 Summary: Add utf8 output to CSV output of overdue_notices.pl Change sponsored?: --- Product: Koha Version: unspecified Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Command-line Utilities Assignee: koha-bugs@lists.koha-community.org Reporter: davewood@gmx.at QA Contact: testopia@bugs.koha-community.org CC: robin@catalyst.net.nz the misc/cronjobs/overdue_notices.pl script produces output/files with confusing encoding apparently depending on the content of the lines written. sometimes with characters encoded in latin1 or utf8 in the same file. https://github.com/Koha-Community/Koha/blob/main/misc/cronjobs/overdue_notic... ``` - open $csv_fh, ">", $csvfilename or die "unable to open $csvfilename: $!"; + open $csv_fh, ">:encoding(UTF-8)", $csvfilename or die "unable to open $csvfilename: $!"; ``` there is possible more needed to fix this then just changing this one line. ``` overdue_notices.pl -v -test -list-all -date="2024-11-01" -csv="overdue.csv" -library="AT-LBST" ``` # examples # ASCII text (only csv headers present) # ISO-8859 text (if german umlauts are present) # Non-ISO extended-ASCII text (if german umlauts AND "č" present) ``` ls 202501*|xargs file 20250101_0310_overdue_test.csv: ASCII text 20250102_0310_overdue_test.csv: ASCII text 20250103_0310_overdue_test.csv: ISO-8859 text 20250104_0310_overdue_test.csv: ASCII text 20250105_0310_overdue_test.csv: ASCII text 20250106_0310_overdue_test.csv: ASCII text 20250107_0310_overdue_test.csv: ISO-8859 text 20250108_0310_overdue_test.csv: ISO-8859 text, with very long lines 20250109_0310_overdue_test.csv: Non-ISO extended-ASCII text, with very long lines 20250110_0310_overdue_test.csv: Non-ISO extended-ASCII text 20250111_0310_overdue_test.csv: ASCII text 20250112_0310_overdue_test.csv: ASCII text 20250113_0310_overdue_test.csv: ISO-8859 text 20250114_0310_overdue_test.csv: Non-ISO extended-ASCII text, with very long lines 20250115_0310_overdue_test.csv: ISO-8859 text, with very long lines 20250116_0310_overdue_test.csv: ISO-8859 text 20250117_0310_overdue_test.csv: ISO-8859 text 20250118_0310_overdue_test.csv: ASCII text 20250119_0310_overdue_test.csv: ASCII text 20250120_0310_overdue_test.csv: ISO-8859 text 20250121_0310_overdue_test.csv: ISO-8859 text 20250122_0310_overdue_test.csv: ISO-8859 text 20250123_0310_overdue_test.csv: ISO-8859 text 20250124_0310_overdue_test.csv: Non-ISO extended-ASCII text, with very long lines 20250125_0310_overdue_test.csv: ASCII text 20250126_0310_overdue_test.csv: ASCII text 20250127_0310_overdue_test.csv: Non-ISO extended-ASCII text 20250128_0310_overdue_test.csv: ISO-8859 text 20250129_0310_overdue_test.csv: ASCII text 20250130_0310_overdue_test.csv: ASCII text 20250131_0310_overdue_test.csv: Non-ISO extended-ASCII text, with very long lines ``` old bugs of the same kind in overdue_notices.pl https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8687 https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26601 -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39023 --- Comment #1 from davewood@gmx.at --- the change i included in the first bug report at least produces a file where german umlaut "Ö" (UTF-8 hex c3 96) and "č" (UTF-8 hex c4 8d) seem to be encoded correctly. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39023 davewood@gmx.at changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|--- |Trivial patch CC| |davewood@gmx.at -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39023 davewood@gmx.at changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |davewood@gmx.at |ity.org | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39023 --- Comment #2 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Hi, do you want to provide the patch file? If you are working with koha-testing docker you can use: git bz attach. If not, a file generated with git format-patch HEAD^ or similar works too. You might want to check the correct name/email address is included in the header lines. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39023 --- Comment #3 from davewood@gmx.at --- Created attachment 177496 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=177496&action=edit patch to set the correct encoding for the CSV output file -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39023 --- Comment #4 from davewood@gmx.at --- Testplan: I) Prepare overdue entry (intranet): 0) Create a patron account with surname Höfler 1) Check if in Patron category administration the overdue column is YES for his code 2) In Tools -> Notice triggers define first delay = 7, letter Overdue notice. Nothing for second and third 3) Check if Tools -> Notices and slips, code overdue is available 4) checkout a book with a retard more than 7 days. 5) Check if Adminstration -> Defining circulation and fine rules for all libraries has fine rules for his patron category, item type book. II) Before applying the patch 0) Run misc/cronjobs/overdue_notices.pl -v -test -list-all -csv="overdue_old.csv" 1) Validate the characters are in Latin1 encoding. (hexdump -C overdue.csv|less) ö => "f6" III) After applying the patch 0) Run misc/cronjobs/overdue_notices.pl -v -test -list-all -csv="overdue_fixed.csv" 1) Validate the characters are in UTF-8 encoding. (hexdump -C overdue_fixed.csv|less) ö => "c3 b6" -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39023 davewood@gmx.at changed: What |Removed |Added ---------------------------------------------------------------------------- Change sponsored?|--- |Sponsored --- Comment #5 from davewood@gmx.at --- Sponsored-by: Styrian State Library -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39023 davewood@gmx.at changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39023 davewood@gmx.at changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #177496|0 |1 is obsolete| | --- Comment #6 from davewood@gmx.at --- Created attachment 177507 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=177507&action=edit add utf8 encoding for csv file output new patch including testplan and sponsored-by statement -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39023 --- Comment #7 from Chris Cormack <chris@bigballofwax.co.nz> --- Automatic patch test results: â ï¸ The patch applies with fuzzy option (ignoring whitespace). This may indicate minor conflicts that need to be resolved. Details: Patch applies with fuzzy option (ignoring whitespace) This is an automated message from the patch testing system. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39023 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39023 --- Comment #8 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 201121 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201121&action=edit Bug 39023: Fix encoding in CSV output file Sponsored-by: Styrian State Library Testplan: I) Prepare overdue entry (intranet): 0) Create a patron account with surname Höfler 1) Check if in Patron category administration the overdue column is YES for his code 2) In Tools -> Notice triggers define first delay = 7, letter Overdue notice. Nothing for second and third 3) Check if Tools -> Notices and slips, code overdue is available 4) checkout a book with a retard more than 7 days. 5) Check if Adminstration -> Defining circulation and fine rules for all libraries has fine rules for his patron category, item type book. II) Before applying the patch 0) Run misc/cronjobs/overdue_notices.pl -v -test -list-all -csv="overdue_old.csv" 1) Validate the characters are in Latin1 encoding. (hexdump -C overdue.csv|less) ö => "f6" III) After applying the patch 0) Run misc/cronjobs/overdue_notices.pl -v -test -list-all -csv="overdue_fixed.csv" 1) Validate the characters are in UTF-8 encoding. (hexdump -C overdue_fixed.csv|less) ö => "c3 b6 Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org