[Bug 25305] New: Double UTF-8 encoding on translation files
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25305 Bug ID: 25305 Summary: Double UTF-8 encoding on translation files Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: I18N/L10N Assignee: koha-bugs@lists.koha-community.org Reporter: bgkriegel@gmail.com QA Contact: testopia@bugs.koha-community.org CC: frederic@tamil.fr Some translation files are showing double UTF-8 encoding, ej. BokmÃ¥l instead of Bokmål Bibliothèque intead of Bibliothèque, and so on -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25305 --- Comment #1 from Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> --- Created attachment 103875 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=103875&action=edit Bug 25305: fix double encoding in translation files Changes inroduced in Bug 24217 and Bug 24365 are producing strings with double encoding Don't know exactly what is producing the problem, but I propose a fix There are two changes, a feature limited use of Modern::Perl and preventing double encoding in output file. To test: 1) Create translation files for a new language ( cd misc/translator; ./translate create xx-YY) 2) Verify double encoding egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-* check strange strings 3) Apply the patch 4) Create po files again ( cd misc/translator; rm -f po/xx-YY*; ./translate create xx-YY) 5) Verify no more double encoding egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-* check normal strings -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25305 Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |24217, 24365 Status|NEW |Needs Signoff Assignee|koha-bugs@lists.koha-commun |bgkriegel@gmail.com |ity.org | Patch complexity|--- |Trivial patch Severity|enhancement |normal Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24217 [Bug 24217] use strict for all modules https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24365 [Bug 24365] Using strict from TmplTokenizer.pm broke the translator script -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25305 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |rel_20_05_candidate -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25305 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #103875|0 |1 is obsolete| | --- Comment #2 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Created attachment 103902 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=103902&action=edit Bug 25305: fix double encoding in translation files Changes inroduced in Bug 24217 and Bug 24365 are producing strings with double encoding Don't know exactly what is producing the problem, but I propose a fix There are two changes, a feature limited use of Modern::Perl and preventing double encoding in output file. To test: 1) Create translation files for a new language ( cd misc/translator; ./translate create xx-YY) A new language means one that isn't already in Koha, xx-YY=>something you invent. 2) Verify double encoding egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-* check strange strings 3) Apply the patch 4) Create po files again ( cd misc/translator; rm -f po/xx-YY*; ./translate create xx-YY) 5) Verify no more double encoding egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-* check normal strings Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25305 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |victor@tuxayo.net --- Comment #3 from Victor Grousset/tuxayo <victor@tuxayo.net> --- (In reply to Bernardo Gonzalez Kriegel from comment #0)
Some translation files are showing double UTF-8 encoding, ej.
BokmÃ¥l instead of Bokmål Bibliothèque intead of Bibliothèque,
and so on
Today I learned that this specific form of encoding issues has a name! Thanks Bernardo :) That might help when facing it in another context. It works :D I added the following note in the test plan:
A new language means one that isn't already in Koha, xx-YY=>something you invent.
-- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25305 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25305 --- Comment #4 from Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> --- Thanks Victor :) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25305 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@bugs.koha-c | |ommunity.org --- Comment #5 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- I am pretty sure this is not the correct fix, we are continuing to hide the true error. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25305 --- Comment #6 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 103906 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=103906&action=edit [ALTERNATIVE-PATCH] Bug 25305: Translation process - Open all files specifying the utf8 encoding -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25305 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |In Discussion --- Comment #7 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to Jonathan Druart from comment #6)
Created attachment 103906 [details] [review] [ALTERNATIVE-PATCH] Bug 25305: Translation process - Open all files specifying the utf8 encoding
What about this patch Bernardo? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25305 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|normal |major -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25305 --- Comment #8 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Note: I missed one occurrence (not used in the normal workflow as it impacts stdout) 368 if (defined $output && $output ne '-') { 369 print STDERR "$0: Opening output file \"$output\"\n" if $verbose_p; 370 open($OUTPUT, '>:encoding(utf-8)', $output) || die "$output: $!\n"; 371 } else { 372 print STDERR "$0: Outputting to STDOUT...\n" if $verbose_p; 373 open($OUTPUT, ">&STDOUT"); 374 } -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25305 --- Comment #9 from Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> --- (In reply to Jonathan Druart from comment #5)
I am pretty sure this is not the correct fix, we are continuing to hide the true error.
True! (In reply to Jonathan Druart from comment #7)
(In reply to Jonathan Druart from comment #6)
Created attachment 103906 [details] [review] [review] [ALTERNATIVE-PATCH] Bug 25305: Translation process - Open all files specifying the utf8 encoding
What about this patch Bernardo?
Way better, I'm only reverting, you are doing something :) I'll test & sign -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25305 Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Discussion |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25305 Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #103906|0 |1 is obsolete| | --- Comment #10 from Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> --- Created attachment 103914 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=103914&action=edit Bug 25305: Translation process - Open all files specifying the utf8 encoding To test: 1) Create translation files for a new language ( cd misc/translator; ./translate create xx-YY) A new language means one that isn't already in Koha, xx-YY=>something you invent. 2) Verify double encoding egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-* check strange strings 3) Apply the patch 4) Create po files again ( cd misc/translator; rm -f po/xx-YY*; ./translate create xx-YY) 5) Verify no more double encoding egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-* check normal string Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Copyied my test plan, can't find more cases, tested using something similar to http://blogs.perl.org/users/chansen/2010/10/coping-with-double-encoded-utf-8... -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25305 Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #103902|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25305 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bgkriegel@gmail.com |jonathan.druart@bugs.koha-c | |ommunity.org --- Comment #11 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Great, thanks Bernardo! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25305 didier <didier.gautheron@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #103914|0 |1 is obsolete| | --- Comment #12 from didier <didier.gautheron@biblibre.com> --- Created attachment 104185 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=104185&action=edit Bug 25305: Translation process - Open all files specifying the utf8 encoding To test: 1) Create translation files for a new language ( cd misc/translator; ./translate create xx-YY) A new language means one that isn't already in Koha, xx-YY=>something you invent. 2) Verify double encoding egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-* check strange strings 3) Apply the patch 4) Create po files again ( cd misc/translator; rm -f po/xx-YY*; ./translate create xx-YY) 5) Verify no more double encoding egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-* check normal string Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Copyied my test plan, can't find more cases, tested using something similar to http://blogs.perl.org/users/chansen/2010/10/coping-with-double-encoded-utf-8... Signed-off-by: Didier Gautheron <didier.gautheron@biblibre.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25305 didier <didier.gautheron@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |didier.gautheron@biblibre.c | |om --- Comment #13 from didier <didier.gautheron@biblibre.com> --- Also tested with a bogus UTF8 file: iconv --verbose -f UTF8 -t ISO_8859-15 koha-tmpl/intranet-tmpl/prog/en/xslt/NORMARCslim2intranetResults.xsl > e.tt mv e.tt koha-tmpl/intranet-tmpl/prog/en/xslt/NORMARCslim2intranetResults.xsl ( cd misc/translator; rm -f po/xx-YY*; ./translate create xx-YY; echo $?) egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-* misc/translator/po/xx-YY-marc-MARC21.po:msgid "Norwegian (Bokmål)" misc/translator/po/xx-YY-marc-NORMARC.po:msgid "Andre typer periodika" misc/translator/po/xx-YY-marc-NORMARC.po:msgid "Artikler (i b\\xF8ker eller periodika)" misc/translator/po/xx-YY-marc-NORMARC.po:msgid "Artikler (i bøker eller periodika)" Of course output is not proper UTF8 but there's no missing lines. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25305 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25305 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #104185|0 |1 is obsolete| | --- Comment #14 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 104226 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=104226&action=edit Bug 25305: Translation process - Open all files specifying the utf8 encoding To test: 1) Create translation files for a new language ( cd misc/translator; ./translate create xx-YY) A new language means one that isn't already in Koha, xx-YY=>something you invent. 2) Verify double encoding egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-* check strange strings 3) Apply the patch 4) Create po files again ( cd misc/translator; rm -f po/xx-YY*; ./translate create xx-YY) 5) Verify no more double encoding egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-* check normal string Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Copyied my test plan, can't find more cases, tested using something similar to http://blogs.perl.org/users/chansen/2010/10/coping-with-double-encoded-utf-8... Signed-off-by: Didier Gautheron <didier.gautheron@biblibre.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25305 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |20.05.00 released in| | Status|Passed QA |Pushed to master -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25305 --- Comment #15 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Nice work everyone! Pushed to master for 20.05 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25305 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |martin.renvoize@ptfs-europe | |.com Keywords|rel_20_05_candidate | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25305 Joy Nelson <joy@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |joy@bywatersolutions.com --- Comment #16 from Joy Nelson <joy@bywatersolutions.com> --- missing dependencies - not backported -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25305 Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=25501 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25305 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |25501 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25501 [Bug 25501] Wide character in print warning installing translations -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org