[Bug 11906] New: Bad display of utf-8 chars in Patron lists (Datatable)
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11906 Bug ID: 11906 Summary: Bad display of utf-8 chars in Patron lists (Datatable) Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: minor Priority: P5 - low Component: Staff Client Assignee: koha-bugs@lists.koha-community.org Reporter: veron@veron.ch QA Contact: testopia@bugs.koha-community.org CC: gmcharlt@gmail.com Created attachment 25916 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=25916&action=edit Bad display of utf-8 chars In patron lists patron names and other texts display ???? if they have umlauts and other utf-8 chars. Something like Беларуская Skjöldóttur dispays as ?????????? Skj�ld�ttur See screenshot attached. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11906 --- Comment #1 from Marc Véron <veron@veron.ch> --- I tried the solution from Bug 11290 In ...patron_lists/list.tt something like: [% p.borrowernumber.surname |html_entity %] It works for Skjöldóttur, but not for Беларуская -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11906 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |In Discussion CC| |tomascohen@gmail.com --- Comment #2 from Tomás Cohen Arazi <tomascohen@gmail.com> --- I'm sorry to say I cannot reproduce the problem right now. I'm pretty sure this is related to some data being defaulted to ISO because of not explicitly marking as UTF8 (so is more related to latin+diacritics than non-latin) and hence breaking places where there is actual UTF-8 data. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11906 --- Comment #3 from Owen Leonard <oleonard@myacpl.org> --- I had a similar problem with patron lists and was pointed to this: http://wiki.koha-community.org/wiki/Koha_on_ubuntu_-_tarball#Tweaking_my.cnf Making that change fixed it for me. Is that the correct solution, or is that a stopgap for a problem we need to address globally? -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11906 --- Comment #4 from Tomás Cohen Arazi <tomascohen@gmail.com> --- (In reply to Owen Leonard from comment #3)
I had a similar problem with patron lists and was pointed to this:
http://wiki.koha-community.org/wiki/Koha_on_ubuntu_-_tarball#Tweaking_my.cnf
Making that change fixed it for me. Is that the correct solution, or is that a stopgap for a problem we need to address globally?
If the problem is environment settings-related (having MySQL set correctly is a pre-requisite for running Koha), this bug should be marked as invalid. I can volunteer to write something to check MySQL configuration sanity so we can (at least) ask users reporting this problems to visit about > system check to see if there's any problem. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11906 Owen Leonard <oleonard@myacpl.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |olli-antti.kivilahti@jns.fi --- Comment #5 from Owen Leonard <oleonard@myacpl.org> --- *** Bug 11942 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11906 Olli-Antti Kivilahti <olli-antti.kivilahti@jns.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |10565 -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11906 Owen Leonard <oleonard@myacpl.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |katrin.fischer@bsz-bw.de --- Comment #6 from Owen Leonard <oleonard@myacpl.org> --- *** Bug 11949 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11906 --- Comment #7 from Marc Véron <veron@veron.ch> --- I followed the advice on http://wiki.koha-community.org/wiki/Koha_on_ubuntu_-_tarball#Tweaking_my.cnf and http://wiki.koha-community.org/wiki/Encoding_and_Character_Sets_in_Koha and added the following to: /etc/mysql/my.cnf ---------- [mysqld] init-connect = 'SET NAMES utf8' character-set-server=utf8 collation-server=utf8_general_ci character_set_client=utf8 ---------- Then I restarted mysql with: /etc/init.d/mysql restart As a result, I get the following in mysql: mysql> show variables like '%colla%'; +----------------------+-----------------+ | Variable_name | Value | +----------------------+-----------------+ | collation_connection | utf8_general_ci | | collation_database | utf8_general_ci | | collation_server | utf8_general_ci | +----------------------+-----------------+ mysql> show variables like '%char%'; +--------------------------+----------------------------+ | Variable_name | Value | +--------------------------+----------------------------+ | character_set_client | utf8 | | character_set_connection | utf8 | | character_set_database | utf8 | | character_set_filesystem | binary | | character_set_results | utf8 | | character_set_server | utf8 | | character_set_system | utf8 | | character_sets_dir | /usr/share/mysql/charsets/ | +--------------------------+----------------------------+ ...and the charecters now display fine. @ Tomás Cohen Arazi Having a MySQL configuration sanity check in about > system check would be great! -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11906 --- Comment #8 from Galen Charlton <gmcharlt@gmail.com> --- Actually... those MySQL configuration changes aren't necessary in order to fix the problem. Something like this suffices: diff --git a/Koha/Database.pm b/Koha/Database.pm index 12758bf..ef1e2db 100644 --- a/Koha/Database.pm +++ b/Koha/Database.pm @@ -55,7 +55,7 @@ sub _new_schema { my $db_passwd = $context->config("pass"); my $schema = Koha::Schema->connect( "DBI:$db_driver:dbname=$db_name;host=$db_host;port=$db_port", - $db_user, $db_passwd ); + $db_user, $db_passwd, { mysql_enable_utf8 => 1 } ); return $schema; } This wouldn't be the final version of this patch, as we wouldn't want to pass msyql_enable_utf8 to a Pg database, and we would have to make sure that binary data (e.g., images stored in the database) don't get mangled, but I think something like this patch would be more reliable. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11906 Galen Charlton <gmcharlt@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Discussion |Needs Signoff -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11906 --- Comment #9 from Galen Charlton <gmcharlt@gmail.com> --- Created attachment 26614 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=26614&action=edit Bug 12001: regression test for using DBIC to store & fetch UTF8 strings This patch adds a regression test for verifying that a DBIx::Class schema object initialized by Koha database sets up the database connection to correct store and retrieve UTF8 values as Perl utf8 strings. To test: [1] Apply this patch. [2] Run prove -v t/db_dependent/Koha_Database.t [3] The test should fail. [4] Apply the main patch for this bug, then do step 2 again. [5] The test should pass. Signed-off-by: Galen Charlton <gmc@esilibrary.com> -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11906 --- Comment #10 from Galen Charlton <gmcharlt@gmail.com> --- Created attachment 26615 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=26615&action=edit Bug 11906: ensure that Koha::Database uses UTF8 mode when connecting to databases This patch fixes an issue whereby the DBIx::Class schema objects were not connecting to the underlying database in UTF8 mode. This most visibility manifested as patron list pages not displaying diacritics correctly. To test: [1] Create a patron list, and make sure that it contains at least one patron whose name or patron category description contains a non-ASCII character. [2] View the list contents; the diacritics should appear mangled. [3] Apply the patch. [4] View the patron list again. This time, the diacritics should be displayed correctly. Note that Apache will also log "list.pl: Wide character in print ...", but this is the lesser of two evils. [5] Verify that prove -v t/db_dependent/Koha_Database.t passes. [6] (extra credit) Verify that t/db_dependent/Koha_Database.t passes when connect to a PostgreSQL database. Signed-off-by: Galen Charlton <gmc@esilibrary.com> -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11906 Chris Cormack <chris@bigballofwax.co.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #26614|0 |1 is obsolete| | --- Comment #11 from Chris Cormack <chris@bigballofwax.co.nz> --- Created attachment 26616 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=26616&action=edit Bug 11906: regression test for using DBIC to store & fetch UTF8 strings This patch adds a regression test for verifying that a DBIx::Class schema object initialized by Koha database sets up the database connection to correct store and retrieve UTF8 values as Perl utf8 strings. To test: [1] Apply this patch. [2] Run prove -v t/db_dependent/Koha_Database.t [3] The test should fail. [4] Apply the main patch for this bug, then do step 2 again. [5] The test should pass. Signed-off-by: Galen Charlton <gmc@esilibrary.com> Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz> -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11906 Chris Cormack <chris@bigballofwax.co.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11906 Chris Cormack <chris@bigballofwax.co.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #26615|0 |1 is obsolete| | --- Comment #12 from Chris Cormack <chris@bigballofwax.co.nz> --- Created attachment 26617 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=26617&action=edit Bug 11906: ensure that Koha::Database uses UTF8 mode when connecting to databases This patch fixes an issue whereby the DBIx::Class schema objects were not connecting to the underlying database in UTF8 mode. This most visibility manifested as patron list pages not displaying diacritics correctly. To test: [1] Create a patron list, and make sure that it contains at least one patron whose name or patron category description contains a non-ASCII character. [2] View the list contents; the diacritics should appear mangled. [3] Apply the patch. [4] View the patron list again. This time, the diacritics should be displayed correctly. Note that Apache will also log "list.pl: Wide character in print ...", but this is the lesser of two evils. [5] Verify that prove -v t/db_dependent/Koha_Database.t passes. [6] (extra credit) Verify that t/db_dependent/Koha_Database.t passes when connect to a PostgreSQL database. Signed-off-by: Galen Charlton <gmc@esilibrary.com> Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz> -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11906 Jonathan Druart <jonathan.druart@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA CC| |jonathan.druart@biblibre.co | |m Blocks| |11944 QA Contact|testopia@bugs.koha-communit |jonathan.druart@biblibre.co |y.org |m --- Comment #13 from Jonathan Druart <jonathan.druart@biblibre.com> --- Galen, We had this patch for bug 11944, so I completely agree with it! Actually it is not on bug 11944, it seems it disappears on rebasing/squashing patches :-/ I still have the patch on a local branch and the difference is: 1/ We added $schema->storage->on_connect_do('SET NAMES utf8'); in Koha::Database::_new_schema Which is stupid! From the mysql doc: "if you turn the flag on after connecting, you will need to issue the command SET NAMES utf8 to get the same effect." 2/ We did not propose unit tests, so ++ :) 3/ We removed the Koha::Template::Plugin::EncodeUTF8 uses in circ/renew.pl This plugin is now useless. I will submit a followup on QAing, feel free to ask open a new specific report for it. Marked as Passed QA. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11906 Jonathan Druart <jonathan.druart@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #26616|0 |1 is obsolete| | Attachment #26617|0 |1 is obsolete| | --- Comment #14 from Jonathan Druart <jonathan.druart@biblibre.com> --- Created attachment 26622 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=26622&action=edit Bug 11906: regression test for using DBIC to store & fetch UTF8 strings This patch adds a regression test for verifying that a DBIx::Class schema object initialized by Koha database sets up the database connection to correct store and retrieve UTF8 values as Perl utf8 strings. To test: [1] Apply this patch. [2] Run prove -v t/db_dependent/Koha_Database.t [3] The test should fail. [4] Apply the main patch for this bug, then do step 2 again. [5] The test should pass. Signed-off-by: Galen Charlton <gmc@esilibrary.com> Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz> Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11906 --- Comment #15 from Jonathan Druart <jonathan.druart@biblibre.com> --- Created attachment 26623 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=26623&action=edit Bug 11906: ensure that Koha::Database uses UTF8 mode when connecting to databases This patch fixes an issue whereby the DBIx::Class schema objects were not connecting to the underlying database in UTF8 mode. This most visibility manifested as patron list pages not displaying diacritics correctly. To test: [1] Create a patron list, and make sure that it contains at least one patron whose name or patron category description contains a non-ASCII character. [2] View the list contents; the diacritics should appear mangled. [3] Apply the patch. [4] View the patron list again. This time, the diacritics should be displayed correctly. Note that Apache will also log "list.pl: Wide character in print ...", but this is the lesser of two evils. [5] Verify that prove -v t/db_dependent/Koha_Database.t passes. [6] (extra credit) Verify that t/db_dependent/Koha_Database.t passes when connect to a PostgreSQL database. Signed-off-by: Galen Charlton <gmc@esilibrary.com> Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz> Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11906 --- Comment #16 from Jonathan Druart <jonathan.druart@biblibre.com> --- Created attachment 26624 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=26624&action=edit Bug 11906: The EncodeUTF8 TT Plugin is useless Now, the data from the DB, return by Koha::Database, have the utf-8 flag turn on. It becomes useless to encode them at the template level. The EncodeUTF8 plugin can be removed too. Test plan: 1/ Check out a book with a title containing non-latin characters. 2/ On circ/renew.pl, renew the item. 3/ In some case, you have a warning box, to override a limit. It should not contain broken strings. If you confirm, you get a "Item renewed" box, it should not contain broken strings. Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11906 Jonathan Druart <jonathan.druart@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|minor |normal -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11906 --- Comment #17 from Jonathan Druart <jonathan.druart@biblibre.com> --- (In reply to Jonathan Druart from comment #13)
3/ We removed the Koha::Template::Plugin::EncodeUTF8 uses in circ/renew.pl This plugin is now useless. I will submit a followup on QAing, feel free to ask open a new specific report for it.
Forget that, it does not work with translated template (the ENCODING flag is not set yet to 'UTF-8' at the template level). -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11906 Jonathan Druart <jonathan.druart@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #26624|0 |1 is obsolete| | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11906 Galen Charlton <gmcharlt@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to Master --- Comment #18 from Galen Charlton <gmcharlt@gmail.com> --- Great minds, and all that. Pushed to master. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11906 Fridolin SOMERS <fridolyn.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to Master |Pushed to Stable CC| |fridolyn.somers@biblibre.co | |m --- Comment #19 from Fridolin SOMERS <fridolyn.somers@biblibre.com> --- Pushed to 3.14.x, will be in 3.14.08 -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11906 Fridolin SOMERS <fridolyn.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|--- |Small patch -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11906 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |13239 -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11906 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks|13239 | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11906 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |13596 -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org