http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #2 from Serhij Dubyk <dubyk@library.lviv.ua> 2011-07-05 15:36:04 UTC --- Created attachment 4561 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=4561 members-home.pl (Russian) -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA Contact for the bug.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #4 from Serhij Dubyk <dubyk@library.lviv.ua> 2011-07-05 15:37:26 UTC --- Created attachment 4563 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=4563 members-home.pl (French) -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA Contact for the bug.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 Serhij Dubyk <dubyk@library.lviv.ua> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|enhancement |critical -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA Contact for the bug.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #6 from Serhij Dubyk <dubyk@library.lviv.ua> 2011-10-18 19:06:54 UTC --- Created attachment 5962 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=5962 -- /cgi-bin/koha/members/members-home.pl (Russian) Tested in Koha 3.05.00.011. -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA Contact for the bug.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #8 from Dobrica Pavlinusic <dpavlin@rot13.org> 2011-10-24 18:11:09 UTC --- Created attachment 6041 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=6041 Clean up Koha utf-8 string handling We are having same problem, but in our case we don't use localized templates but have utf-8 characters inside MySQL which get double utf-8 encoded before they are sent to browser. If you feel brave, you can try attached patch with fixes problem for us. Below is short summary of changes from patch: This patch tries to clean up utf-8 handling in Koha. In current implementation (mostly commented out in this patch) uses heuristic to guess which strings need decoding from utf-8 to binary representation and doesn't support utf-8 characters in templates and has problems with utf-8 data from database. With this changes, Koha perl code always uses utf-8 encoding correctly. All incomming data from database is allready correctly marked as utf-8, and decoding of utf8 is required only from Zebra and XSLT transfers which don't set utf-8 flag correctly. For output, standard perl :utf8 handler is used removing various "wide character" warnings as side-effect. -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA Contact for the bug.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #10 from Paul Poulain <paul.poulain@biblibre.com> 2011-10-26 16:55:48 UTC --- Created attachment 6065 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=6065 encoding problem in member-home.pl in 3.6.0 (another one) -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA Contact for the bug.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #12 from Serhij Dubyk <dubyk@library.lviv.ua> 2011-10-27 08:06:10 UTC --- Created attachment 6070 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=6070 member.pl (Ukrainian) after patch1 -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA Contact for the bug.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #14 from Serhij Dubyk <dubyk@library.lviv.ua> 2011-10-27 08:09:05 UTC --- Created attachment 6072 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=6072 Staff home page (Ukrainian) after patch1 -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA Contact for the bug.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #16 from Serhij Dubyk <dubyk@library.lviv.ua> 2011-10-27 14:41:05 UTC --- The result of testing the second patch. I checked a few dozen pages opac and staff and saw: 1) pages of patron home andd search now looks good; 2) most of the tested pages too good; 3) a few pages or places where there are still problems - a) the library name on each page; b) search result in the catalog (opac and staff); c) page modify server Z39-50; d) page with system parameters (descriptions from ..pref..po) (also added screenshots for this) -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA Contact for the bug.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #18 from Serhij Dubyk <dubyk@library.lviv.ua> 2011-10-27 14:43:21 UTC --- Created attachment 6077 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=6077 STAFF search (Ukrainian, after patch 2) -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA Contact for the bug.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #20 from Serhij Dubyk <dubyk@library.lviv.ua> 2011-10-27 14:45:30 UTC --- Created attachment 6079 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=6079 z39-50 edit page (ukrainian, after patch 2) -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA Contact for the bug.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #22 from Dobrica Pavlinusic <dpavlin@rot13.org> 2011-10-27 19:29:24 UTC --- Created attachment 6083 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=6083 added session, CGI queries and all Zebra/PAR utf8 decode Thanks for report. Attached patch adds: - encoding for data from session (like library name) - utf8 force for CGI queries which include utf-8 text[1] - decode all utf8 results from Zebra/PAR 1: I would guess that I missed some of pages where utf-8 text is posted to Koha, but setting it everywhere would break upload according to CGI docs, so I'm conservative here. -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA Contact for the bug.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #24 from Serhij Dubyk <dubyk@library.lviv.ua> 2011-10-27 22:07:34 UTC --- Created attachment 6084 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=6084 opac_search.pl (ukrainian, after patch 3) -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA Contact for the bug.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #26 from Dobrica Pavlinusic <dpavlin@rot13.org> 2011-10-28 09:50:53 UTC --- This is very strange. Do tables in your database use utf-8 encoding? Could you dump your Koha database somewhere and mail me link to dpavlin@rot13.org so I can test with your data? I already installed languages, so this would help me in debugging without bothering you for each update. -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA Contact for the bug.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |m.de.rooy@rijksmuseum.nl --- Comment #28 from M. de Rooy <m.de.rooy@rijksmuseum.nl> 2011-11-02 13:49:23 UTC --- Just some "fast" preliminary QA remarks from glimsing through the patches. I agree that encoding data in Template.pm is actually not correct and that specifying utf8 on std output with binmode or use open is a better practice than what is done in current Koha. Check showmarc.p/opac-showmarc.pl where I recently decoded output from template to prevent double encoding. This is a place that therefore is touched also by your patch. Another recent bug report pointed out that using :encoding(utf8) is preffered above :utf8. The latter form is a lazy one in Perl. I saw that you decode output from the Zoom connection in Search.pm. Without further investigation I am inclined to think that this is not (always) correct. Apparently, you assume that the output from Zoom is utf8 octets. Isn't it Latin and doesn't Zoom allow you to pass a charset parameter like charset=latin,utf8 in order to change encoding? Could you verify? -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA Contact for the bug.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 Dobrica Pavlinusic <dpavlin@rot13.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #6041|0 |1 is obsolete| | -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA Contact for the bug. You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 Dobrica Pavlinusic <dpavlin@rot13.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #6073|0 |1 is obsolete| | -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA Contact for the bug. You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |katrin.fischer@bsz-bw.de Patch Status|Needs Signoff |Does not apply --- Comment #30 from Katrin Fischer <katrin.fischer@bsz-bw.de> 2011-12-21 06:35:36 UTC --- It seems there is something wrong with the attached patch: [~/kohaclone (nso_6554)]> git bz apply 6554 Bug 6554 - Broken encoding in members home/search page added session, CGI queries and all Zebra/PAR utf8 decode Apply? [yn] y Patch format detection failed. Patch left in /tmp/added-session-CGI-queries-and-all-ZebraPAR-utf8-de-iXfemD.patch -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA Contact for the bug. You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 MJ Ray (software.coop) <mjr@software.coop> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #6083|0 |1 is obsolete| | --- Comment #32 from MJ Ray (software.coop) <mjr@software.coop> --- Created attachment 10118 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=10118&action=edit Bug 6554 - added session CGI queries and all ZebraPAR utf8_decode Rebased from added-session-CGI-queries-and-all-ZebraPAR-utf8-de-O_t1Z2.patch Original-Author: Dobrica Pavlinusic <dpavlin@rot13.org> -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|PATCH-Sent (DO NOT USE) |P3 --- Comment #33 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I installed uk-UA (Ukrainian) and tried to replicate the display problems. I checked the item type page, creating new itemtypes with "ukrainian" (copied some strings from other places) descriptions, I checked the Z39.50 page and the system preferences. I am not sure how to test this, without being able to replicate the problem. Dobrica, can you help? -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |http://bugs.koha-community. | |org/bugzilla3/show_bug.cgi? | |id=9772 -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 Dobrica Pavlinusic <dpavlin@rot13.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Discussion |Needs Signoff Patch complexity|--- |Medium patch -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 Dobrica Pavlinusic <dpavlin@rot13.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #16780|0 |1 is obsolete| | --- Comment #36 from Dobrica Pavlinusic <dpavlin@rot13.org> --- Created attachment 16797 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=16797&action=edit Bug 6554 - make Koha internally utf-8 clean In current implementation (mostly commented out in this patch) uses heuristic to guess which strings need decoding from utf-8 to binary representation and doesn't support utf-8 characters in templates and has problems with utf-8 data from database. With this changes, Koha perl code always uses utf-8 encoding correctly. All incomming data from database is allready correctly marked as utf-8, and decoding of utf8 is required only from Zebra and XSLT transfers which don't set utf-8 flag correctly. For output, standard perl :encoding(utf8) handler is used so it also removes various "wide character" warnings as side-effect. Test scenario: 1. make sure that you have utf-8 characters in your biblio records, patrons, categories etc. 2. try to search records on intranet and opac which contain utf-8 characters 3. install language which has utf-8 characters, e.g. uk-UA dpavlin@koha-dev:/srv/koha/misc/translator(bug_6554) $ PERL5LIB=/srv/koha/ perl translate install uk-UA 4. switch language to uk-UA and verify that templates display correctly 5. test search and Z39.50 search and verify that caracters are correct This version uses more strict :encoding(utf8) and removes unneeded decode of XSLT which comes back as correctly encoded strings. -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 Owen Leonard <oleonard@myacpl.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #16797|0 |1 is obsolete| | --- Comment #37 from Owen Leonard <oleonard@myacpl.org> --- Created attachment 16857 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=16857&action=edit [SIGNED-OFF] Bug 6554 - make Koha internally utf-8 clean In current implementation (mostly commented out in this patch) uses heuristic to guess which strings need decoding from utf-8 to binary representation and doesn't support utf-8 characters in templates and has problems with utf-8 data from database. With this changes, Koha perl code always uses utf-8 encoding correctly. All incomming data from database is allready correctly marked as utf-8, and decoding of utf8 is required only from Zebra and XSLT transfers which don't set utf-8 flag correctly. For output, standard perl :encoding(utf8) handler is used so it also removes various "wide character" warnings as side-effect. Test scenario: 1. make sure that you have utf-8 characters in your biblio records, patrons, categories etc. 2. try to search records on intranet and opac which contain utf-8 characters 3. install language which has utf-8 characters, e.g. uk-UA dpavlin@koha-dev:/srv/koha/misc/translator(bug_6554) $ PERL5LIB=/srv/koha/ perl translate install uk-UA 4. switch language to uk-UA and verify that templates display correctly 5. test search and Z39.50 search and verify that caracters are correct Signed-off-by: Owen Leonard <oleonard@myacpl.org> I followed the test plan, adding utf-8 characters to library names, patron categories, titles, and authorized values. I tried the uk-UA translation and everything looked good. When performing Z39.50 searches for titles containing utf-8 characters I got results which were still occasionally contaminated with dummy characters [?] but I assume this is Z39.50's fault not the patch's. -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #39 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- This would work for me: Output.pm (binmode removed), output_with_http_headers: $data =~ s/\&\;amp\; /\&\; /g; utf8::encode($data); print $query->header($options), $data; XSLT.pm (XSLTParse4Display): my $newxmlrecord = $stylesheet->{$xslfilename}->output_as_chars($results); ? utf8::decode($newxmlrecord); return $newxmlrecord; Note that output_string seems to be deprecated. Decoding newxmlrecord seems to be not necessary.. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #41 from Dobrica Pavlinusic <dpavlin@rot13.org> --- Created attachment 16905 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=16905&action=edit Bug 6554 - followup - encode output, XSLT encoding from templates Encode output data instead of using binmode to be more compliant with various usage (e.g. plack). XSLT output_string seems to be depriciated call, decoding of newxmlrecord seems to be not necessary. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #16905|0 |1 is obsolete| | --- Comment #43 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 16929 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=16929&action=edit Bug 6554 QA Followup Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Removing binmode, now encoding data in output_with_http_headers. Replaced output_string by output_as_chars in XSLTParse4Display. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #45 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 16930 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=16930&action=edit Merge branch 'bug_9919' into 3.12-master Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de> Simple fix for a bad problem. Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com> -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #47 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 16931 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=16931&action=edit Bug 6554 - make Koha internally utf-8 clean In current implementation (mostly commented out in this patch) uses heuristic to guess which strings need decoding from utf-8 to binary representation and doesn't support utf-8 characters in templates and has problems with utf-8 data from database. With this changes, Koha perl code always uses utf-8 encoding correctly. All incomming data from database is allready correctly marked as utf-8, and decoding of utf8 is required only from Zebra and XSLT transfers which don't set utf-8 flag correctly. For output, standard perl :encoding(utf8) handler is used so it also removes various "wide character" warnings as side-effect. Test scenario: 1. make sure that you have utf-8 characters in your biblio records, patrons, categories etc. 2. try to search records on intranet and opac which contain utf-8 characters 3. install language which has utf-8 characters, e.g. uk-UA dpavlin@koha-dev:/srv/koha/misc/translator(bug_6554) $ PERL5LIB=/srv/koha/ perl translate install uk-UA 4. switch language to uk-UA and verify that templates display correctly 5. test search and Z39.50 search and verify that caracters are correct Signed-off-by: Owen Leonard <oleonard@myacpl.org> I followed the test plan, adding utf-8 characters to library names, patron categories, titles, and authorized values. I tried the uk-UA translation and everything looked good. When performing Z39.50 searches for titles containing utf-8 characters I got results which were still occasionally contaminated with dummy characters [?] but I assume this is Z39.50's fault not the patch's. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #16929|0 |1 is obsolete| | --- Comment #48 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 16932 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=16932&action=edit Bug 6554 QA Followup Removing binmode, now encoding data in output_with_http_headers. Replaced output_string by output_as_chars in XSLTParse4Display. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #50 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Acquisitions: look for é in the vendor field. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 Jonathan Druart <jonathan.druart@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@biblibre.co | |m --- Comment #52 from Jonathan Druart <jonathan.druart@biblibre.com> --- (In reply to comment #51)

(In reply to comment #50)

...

Acquisitions: look for é in the vendor field. Order search too.

Administration: look for pref é. Does not exist, but encoding error.

That makes them all for a short tour through the staff client (with comment 44, 49 and 50). Please check if you can find another one :)

I didn't really dive into this issue, but encoding errors don't appears without the last patch ("QA Followup"), at least for member and syspref search. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #54 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to comment #53)

(In reply to comment #52)

...

I didn't really dive into this issue, but encoding errors don't appears without the last patch ("QA Followup"), at least for member and syspref search.

I will check it too.

Jonathan had a line commented out. So this is no issue. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #56 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 16961 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=16961&action=edit Bug 6554 Followup for circulation patron search Adds decoding for patron name. Test plan: Go to Circulation. Enter a name with a diacritic. Find that patron or get a message with the name you entered without converted characters. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #58 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 16963 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=16963&action=edit Bug 6554: Followup for serial search Adds decoding for title, publisher and vendor. Test plan: Go to serials. Search for a diacritic in title. Check. Go to Advanced search (in Serials). Search for diacritic in title, vendor or publisher. Check. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff --- Comment #60 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- I resolved the points that i found. This should still receive some signoff. And some people willing to check if they can still find another problem with url parameter handling.. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #16962|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #16966|0 |1 is obsolete| | --- Comment #63 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 16968 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=16968&action=edit Bug 6554: Followup for preferences.pl Decodes searchfield. Test plan: Look for e accent in preferences. You should no longer see converted chars. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #65 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Resubmitted the followups for acq and prefs. They did not need URI::Escape after all. CGI handles that correctly. Simplification :) -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #16931|0 |1 is obsolete| | --- Comment #67 from Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> --- Created attachment 16984 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=16984&action=edit [SIGNED-OFF] Bug 6554 - make Koha internally utf-8 clean In current implementation (mostly commented out in this patch) uses heuristic to guess which strings need decoding from utf-8 to binary representation and doesn't support utf-8 characters in templates and has problems with utf-8 data from database. With this changes, Koha perl code always uses utf-8 encoding correctly. All incomming data from database is allready correctly marked as utf-8, and decoding of utf8 is required only from Zebra and XSLT transfers which don't set utf-8 flag correctly. For output, standard perl :encoding(utf8) handler is used so it also removes various "wide character" warnings as side-effect. Test scenario: 1. make sure that you have utf-8 characters in your biblio records, patrons, categories etc. 2. try to search records on intranet and opac which contain utf-8 characters 3. install language which has utf-8 characters, e.g. uk-UA dpavlin@koha-dev:/srv/koha/misc/translator(bug_6554) $ PERL5LIB=/srv/koha/ perl translate install uk-UA 4. switch language to uk-UA and verify that templates display correctly 5. test search and Z39.50 search and verify that caracters are correct Signed-off-by: Owen Leonard <oleonard@myacpl.org> I followed the test plan, adding utf-8 characters to library names, patron categories, titles, and authorized values. I tried the uk-UA translation and everything looked good. When performing Z39.50 searches for titles containing utf-8 characters I got results which were still occasionally contaminated with dummy characters [?] but I assume this is Z39.50's fault not the patch's. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Already signed, add mine. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #16963|0 |1 is obsolete| | --- Comment #69 from Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> --- Created attachment 16986 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=16986&action=edit [SIGNED-OFF] Bug 6554: Followup for serial search Adds decoding for title, publisher and vendor. Test plan: Go to serials. Search for a diacritic in title. Check. Go to Advanced search (in Serials). Search for diacritic in title, vendor or publisher. Check. Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Works ok, no errors. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #16968|0 |1 is obsolete| | --- Comment #71 from Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> --- Created attachment 16988 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=16988&action=edit [SIGNED-OFF] Bug 6554: Followup for preferences.pl Decodes searchfield. Test plan: Look for e accent in preferences. You should no longer see converted chars. Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Works ok, no errors. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off CC| |bgkriegel@gmail.com --- Comment #73 from Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> --- I found that this solves the problem reported in Bug 9579. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 Karam Qubsi <karamqubsi@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |karamqubsi@gmail.com -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 mathieu saby <mathieu.saby@univ-rennes2.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mathieu.saby@univ-rennes2.f | |r --- Comment #76 from mathieu saby <mathieu.saby@univ-rennes2.fr> --- Thank you Bernardo! From the title of this bug, I could never imagine it solved the problem of arabic facets... Mathieu Saby -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Component|Patrons |Architecture, internals, | |and plumbing -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #79 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- I also tested this with record containing Chinese characters. No problem with displaying the record, editing and saving. Dobrica: Note that some decodes are not always visible in the script. Addbiblio for instance 'hides' them in the Biblio module: TransformHtmlToMarc. Consistency in Koha is a large and continuing project :) -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #80 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- QA Comment: This report is pending for some time now. Dobrica's change (incl. the first followup) is a great fix for UTF-8 handling in template processing. We make Koha adhere to "Decode what goes in, encode what goes out". It is a welcome fix for non-English languages with diacritics. But it does have quite some impact. All occurrences where cgi parameters with utf8 chars not being decoded, will show up. Note that you could argue that these place are actually existing bugs that were hidden by the templates bug. My strong recommendation is nevertheless to push this patch. We have repaired the most important and visible occurrences of wrong parameter handling. I opened up a new report, bug 9951, to add new ones. I checked the staff client via interface and went through all opac scripts. With tests from Owen, Dobrica, Bernardo and myself with various languages and characters, this patch has a solid base. So, have courage, RM :) Code looks good to me. qa script does not complain. Passed QA -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 Dobrica Pavlinusic <dpavlin@rot13.org> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |http://bugs.koha-community. | |org/bugzilla3/show_bug.cgi? | |id=9579 -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 Colin Campbell <colin.campbell@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |colin.campbell@ptfs-europe. | |com --- Comment #82 from Colin Campbell <colin.campbell@ptfs-europe.com> --- This patch breaks Koha.. On login you receive Modification of a read-only value attempted at /home/cc/kohaclone/C4/Context.pm line 1107. Problem is the call to decode within map $_ is an alias to a value which may be read-only -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #84 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 17179 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=17179&action=edit Bug 6554: Followup for decoding in userenv Thanks to Colin. He reported this problem: Modification of a read-only value attempted at C4/Context.pm line 1107. $_ is an alias to a value which may be read-only in the map call. Could not reproduce this error, but this should prevent it. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #86 from Colin Campbell <colin.campbell@ptfs-europe.com> --- I think the problem is with perls greater than 5.10 the map documentation suggests you should use a loop in this case. I' test and let you know -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #88 from Paul Poulain <paul.poulain@biblibre.com> --- (In reply to comment #86)

I think the problem is with perls greater than 5.10 the map documentation suggests you should use a loop in this case. I' test and let you know

could it be a problem of a login with mysql user/password ? There are problems/errors like this sometimes with the mysql login, that is borrowernumber=0 and many other infos missing. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #90 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to comment #88)

could it be a problem of a login with mysql user/password ? There are problems/errors like this sometimes with the mysql login, that is borrowernumber=0 and many other infos missing.

I had no problems with the install login.. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #92 from Colin Campbell <colin.campbell@ptfs-europe.com> --- Created attachment 17186 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=17186&action=edit Proposed Patch I've tested Marcel's patch and it works...however I've also sent my own version which removes the map. The reason is that multi-statement blocks in maps can lead to maintenance headaches and are a bit harder to track down if an error occurs (perlcritic on stricter settings complains). The map doc suggests a loop for this case. I've kept the comment in but on CGI::Session's homepage the utf8 bugs are marked as resolved -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #17179|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #16985|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #16987|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #16989|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #94 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 17187 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=17187&action=edit Bug 6554 Fix error caused by modifying $_ in a map Error 'Modification of a read-only value attempted' triggered on login because of manipulation of $_ in the map Moved the mod to a loop as recommended in the doc for map Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA --- Comment #95 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- QA Comment: Patch of Colin is a followup rewriting the map statement introduced here in setuserenv (Context.pm). Several people could not reproduce the problem Colin reported. But this is a code improvement and fixes his problem. Passed QA -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #97 from Karam Qubsi <karamqubsi@gmail.com> --- Created attachment 17327 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=17327&action=edit NonLatin user name on Koha pages This is how the name كرم appears -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #99 from Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> --- I can reproduce this problem using Karam's name. What I found so far is: 1) Fresh login, session cookie (in DB) records correctly userid id: كر 2) click on logo (or any link) and now session cookie has userid mangled id: "\xD9\x83\xD8\xB1\xD9\x85" 3) click any page, same result 4) Go to Administration > System Preferences, and now session cookie has correct userid again (also loggedinusername) So, could be something related with encoding at cookie writing time. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #17327|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |http://bugs.koha-community. | |org/bugzilla3/show_bug.cgi? | |id=10019 -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |http://bugs.koha-community. | |org/bugzilla3/show_bug.cgi? | |id=10074 -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 Jared Camins-Esakov <jcamins@cpbibliography.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |10019, 10074 -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 Karam Qubsi <karamqubsi@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |http://bugs.koha-community. | |org/bugzilla3/show_bug.cgi? | |id=10128 -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 Galen Charlton <gmcharlt@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to Master |In Discussion --- Comment #103 from Galen Charlton <gmcharlt@gmail.com> --- Following Jared, I have also reverted the patches for this bug (and related bugs) from master. This should not be viewed as a permanent reversion, although I have reservations about turning on CGI.pm's -utf8 switch across the board -- but I have even stronger reservations about the ensuant game of whack-a-mole we've been playing with new bugs arising from this patch. By next week I will put together a separate integration branch to use as a basis for testing this more thoroughly. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |http://bugs.koha-community. | |org/bugzilla3/show_bug.cgi? | |id=10114 -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #106 from Galen Charlton <gmcharlt@gmail.com> --- (In reply to comment #104)

Here is my reaction on this action from you as new RM:

I am also an _old_ RM, and stand by my decision on the following basis: - This was reverted in 3.12.x. Jared would have been perfectly in his rights to revert in master as well. If the patch series was too problematic to include in the main release, it does not necessarily belong in master, either. - Side-effects of this patch series were *getting in the way* of applying other patches necessary for the 3.12.x release. - The basis of the patch series relies on use of a flag to CGI.pm that it's own documentation warns against. Further, please bear in mind what I said: "This should not be viewed as a permanent reversion" And I mean it. I do not disagree that the underlying bug needs to be dealt with, but the number of side-effects shows that the current patch series is not ready for prime time. The integration branch will provide an opportunity for more thorough testing.

On the whole: This is very discouraging and highly unmotivating. Please communicate with other developers and team members instead of starting your RM term with one-sided actions like this one. Do you think that this makes me spend more time on the subject?

Considering the discussion and advance warning over IRC, I reject your implication that this decision was not communicated. The reversion is not meant as a personal slight against anybody. I do understand that you have put a lot of effort into this, and I understand that having patches that one has worked on getting reverted stings. I hope to encourage a different approach to patches and reversions: that they be treated as experiments and learning opportunities, and that credit be given to those who make the experiment, even if their patch doesn't end up getting committed in its original form. However, I acknowledge that this is not an easy thing. But I will also be blunt: I would not be doing my job as RM well if, when necessary, I didn't exercise my own judgment. One of the things I've learned the hard way is that at times, it is necessary to force a pause to recheck assumptions. I believe this is one of the moments for this bug. I encourage you to work on the integration branch, but if you prefer to take a break from this bug, I have every confidence that we will not lack for hands and eyes to test. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #108 from Dobrica Pavlinusic <dpavlin@rot13.org> --- (In reply to comment #103)

This should not be viewed as a permanent reversion, although I have reservations about turning on CGI.pm's -utf8 switch across the board -- but I have even stronger reservations about the ensuant game of whack-a-mole we've been playing with new bugs arising from this patch.

Would you mind to elaborate your objections to CGI.pm's -utf8 switch? I think that it's cleaner to take extra care about code which uses file upload than to hand-decode every single parameter we are passing into Koha code. This is only my personal preference, and I am fine with whatever solution gets eventually prefered, but I main question is if I should re-submit my patches without -utf8 in them. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #110 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to comment #105)

This may be the only time in bugzilla history where a link to a video of a game is appropriate. :-)

(In reply to comment #106) Thanks for your reply, but I still disagree. Nevertheless I propose to end that discussion here.. (In reply to comment #107)

I don't know why, a remainder of the first patch still exists on master, which cause some trouble with encoding. Communicated this with Jared on bug 10070.

-- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #17187|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |tomascohen@gmail.com -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #113 from Serhij Dubyk <dubyk@library.lviv.ua> --- Created attachment 18746 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=18746&action=edit Search for preferences in Cyrillic -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #115 from Serhij Dubyk <dubyk@library.lviv.ua> --- Created attachment 18749 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=18749&action=edit Members Home -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #117 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to comment #112)

In Koha 3.12 some encoding problems was returned. Currently found in 3 locations: http://localhost:8888/cgi-bin/koha/members/members-home.pl (main screen) http://localhost:8888/cgi-bin/koha/members/member.pl (search) http://localhost:8888/cgi-bin/koha/circ/circulation.pl (search and menu) http://localhost:8888/cgi-bin/koha/admin/preferences.pl (search for non-latin) Screenshots attached. Search by members completely can not be used in 3.12.0.

Horrible pictures, Serhij. But I still think that we were on the right track with Dobrica's patch to make Koha internally utf8 clean. Such a change can be compared with the template engine change we had some years ago. You cannot solve every little problem after that in one run. But the wind blew in another direction :) -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 Josef Moravec <josef.moravec@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |josef.moravec@gmail.com -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 skobyanka@mail.ru <skobyanka@mail.ru> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |skobyanka@mail.ru --- Comment #120 from skobyanka@mail.ru <skobyanka@mail.ru> --- (In reply to Mike from comment #119)

I found same problem in 3.12.x in Czech translation. Problem on: /cgi-bin/koha/members/members-home.pl /cgi-bin/koha/admin/preferences.pl?tab=circulation /cgi-bin/koha/admin/preferences.pl?tab=opac /cgi-bin/koha/admin/preferences.pl?tab=patrons

I solved the problem in /cgi-bin/koha/members/members-home.pl by removing from the Advanced Search patron Search by category: /usr/share/koha/intranet/htdocs/intranet-tmpl/prog/ru-RU/includes/patron-search.inc Delete: [% IF ( categories ) %] <p><label for="categorycode">Категория: </label><select name="categorycode" id="categorycode"> <option value="">Произвольная</option>[% FOREACH categorie IN categories %] [% IF ( categorie.selected ) %] <option value="[% categorie.categorycode %]" selected="selected">[% categorie.description %]</option>[% ELSE %] <option value="[% categorie.categorycode %]">[% categorie.description %]</option>[% END %] [% END %]</select></p> [% END %] -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 Ketan Kulkarni <ketan@carvingit.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Discussion |Needs Signoff Assignee|dpavlin@rot13.org |ketan@carvingit.com --- Comment #122 from Ketan Kulkarni <ketan@carvingit.com> --- Created attachment 21318 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=21318&action=edit An update to Templates.pm The patch is made against tag v3.12.04. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 Ketan Kulkarni <ketan@carvingit.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #125 from Ketan Kulkarni <ketan@carvingit.com> --- Hello Chris, Hello Chris, I am sorry, I did not know the process. I did think it was a little odd to sign-off myself. However, the message said "Patch applied cleanly, go forth and signoff" so I did it. Anyway, thanks for the information. Here's the test plan. * Enter Patron names in non-latin characters like Russian. There is a screenshot of submitted by a user earlier. That is in Russian. In my case, the patron names are in Devnagari. * Then carry out a search on Patron * It should show you Patron names as they were entered. * At the same time, other parts of the site like - Catalog search should work with unicode. * A failure scenario is when the results show strange characters on the web page. I'll work on testing others' patches too. I am a little new to the whole process yet. Regards, Ketan (In reply to Chris Cormack from comment #124)

Hi Ketan

I appreciate your enthusiasm, and we definitely want more people submitting patches. But unfortunately you can't mark your own patch signed off.

Here's a quick write up on how it works http://wiki.koha-community.org/wiki/Bug-enhancement-patch_Workflow

I would be happy to try signing off on your patch. Can you give me a quick test plan of how to recreate the problem so I can test the patch fixes it?.

And of course we would love it if you tested and signed off other's patches.

Thanks

Chris

-- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #127 from Ketan Kulkarni <ketan@carvingit.com> --- Hello Chris, Here are a couple strings. Try them in Patron search of 3.12.4. कुलकर्णी पाटील The above are from devnagari script used for Marathi language. Regards, Ketan (In reply to Chris Cormack from comment #126)

Hi

I'm having trouble recreating the problem, I can enter cyrillic characters and have them display fine without the patch.

So I can't really test that it fixes the issue, as I can't make the bad characters show.

Could you paste a string that breaks it for you?

-- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #129 from Ketan Kulkarni <ketan@carvingit.com> --- Ah yes. It breaks the dropdown of New Patron. Quite a few things have changed in the master after v3.12.4. I'll check the drop down part. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #131 from Chris Cormack <chris@bigballofwax.co.nz> --- (In reply to Josef Moravec from comment #130)

Created attachment 21399 [details] members-home czech

Instead of adding patron with non-latin characters in name, try to define patron categories with non-latin characters. It definitely breaks the members-home page for me.

Excellent, thanks for that. I'll wait for a new patch and test it that way. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #133 from Ketan Kulkarni <ketan@carvingit.com> --- Created attachment 21478 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=21478&action=edit Double encoding utf8 problem Use of utf8::upgrade() instead of utf8::encode() appears to be working. While taking inputs, they should be decoded first whereas output should always be encoded. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #134 from Ketan Kulkarni <ketan@carvingit.com> --- I have another patch ready which is an enhancement over the earlier patch I submitted. How can I discard the older patch and add a new one? -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 Ketan Kulkarni <ketan@carvingit.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #21478|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 Josef Moravec <josef.moravec@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #21318|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #138 from Ketan Kulkarni <ketan@carvingit.com> --- Hello Joseph, Could you please attach a screenshot the broken "edit patron categories" page. I am unable to reproduce the problem after the latest update. i.e. the last patch. Screenshot of any other page showing the problem will also do. Regards, Ketan (In reply to Josef Moravec from comment #137)

I've just tried the last patch in sandbox, but unfortunately, it doesn't work for me. In fact, there are another pages broken after applying patch. (e.g. edit patron categories page)

-- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #140 from Josef Moravec <josef.moravec@gmail.com> --- I've just uploaded the screenshot. But there is a weird issue... After night, on the same sandbox setup, the problem with catron categories somehow disappeared, but after new setup, with the same parameters, the problem again appeared. Problem on the members home page is still the same. (In reply to Ketan Kulkarni from comment #138)

Hello Joseph,

Could you please attach a screenshot the broken "edit patron categories" page. I am unable to reproduce the problem after the latest update. i.e. the last patch.

Screenshot of any other page showing the problem will also do.

Regards, Ketan

(In reply to Josef Moravec from comment #137)

...

I've just tried the last patch in sandbox, but unfortunately, it doesn't work for me. In fact, there are another pages broken after applying patch. (e.g. edit patron categories page)

-- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #142 from Ketan Kulkarni <ketan@carvingit.com> --- Created attachment 21509 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=21509&action=edit The template problem with utf8 For non-latin templates, almost all pages need to be seen along with the patron categories page. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 Ketan Kulkarni <ketan@carvingit.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #21488|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 Srdjan Jankovic <srdjan@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #21581|0 |1 is obsolete| | --- Comment #144 from Srdjan Jankovic <srdjan@catalyst.net.nz> --- Created attachment 21629 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=21629&action=edit bug 6554: double utf8 encoding fix * added ENCODING=>utf8 to Template->new * added utf8::decode for ms_query -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #146 from Ketan Kulkarni <ketan@carvingit.com> --- Thank you Srdjan for consolidating the patch and also for removing the if() from utf8::upgrade($some_var) if utf8::is_utf8($some_var); The condition is not needed. Regards, Ketan (In reply to Srdjan Jankovic from comment #145)

I cannot sign off, it has tabs instead of spaces.

I have replaced the multi-patch with this consolidated one. Please check if the commit message is ok.

Also, I don't have time to look deeper, but utf8::upgrade($some_var) if utf8::is_utf8($some_var); looks wrong, that should not be needed. Or at least looks like the problem is being addressed in the wrong place.

ENCODING => 'utf8' makes sense.

Encode::decode() would also makes sense I presume.

-- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #148 from Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> ---

How do I remove tabs and put spaces? I use vi editor and I use tab for indentation.

Don't use tabs if you expect to pass QA http://stackoverflow.com/questions/426963/replace-tab-with-spaces-in-vim -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 Srdjan Jankovic <srdjan@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 sandboxes@biblibre.com <sandboxes@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |sandboxes@biblibre.com --- Comment #151 from sandboxes@biblibre.com <sandboxes@biblibre.com> --- Patch tested with a sandbox, by Josef Moravec <josef.moravec@gmail.com> -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #152 from sandboxes@biblibre.com <sandboxes@biblibre.com> --- Created attachment 21682 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=21682&action=edit UTF8 double encoding fixes. Commits squashed into one. double utf8 encoding fix .. second try minor formatting updates enhancement to the previous patch related to utf8 double encoding added ENCODING=>utf8 to new-> added space which is there in master added utf8::decode for ms_query replaced tabs with spaces replaced tabs with spaces removed extra space Signed-off-by: Srdjan <srdjan@catalyst.net.nz> http://bugs.koha-community.org/show_bug.cgi?id=6554 Signed-off-by: Josef Moravec <josef.moravec@gmail.com> -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #153 from Tomás Cohen Arazi <tomascohen@gmail.com> --- I marked 10068 as 'see also', not because of the patches, but to mention the first comment. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #155 from Ketan Kulkarni <ketan@carvingit.com> --- Hello Tomás, Yes, we should move from utf8::encode/decode to Encode:: This particular bug (6554) is about double encoding that happens in Templates.pm. I guess we should treat 10068 as a separate thing. It is linked for sure with the same issue. However, it's more about encoding. This one deals with decoding. Ketan (In reply to Tomás Cohen Arazi from comment #153)

I marked 10068 as 'see also', not because of the patches, but to mention the first comment.

-- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #157 from Ketan Kulkarni <ketan@carvingit.com> --- Hello, Here's the scenario to reproduce the error condition. * Create patron categories with non-latin characters and then perform a search on patrons using non-latin characters. The patch is trying to fix two things. * Double encoding of utf8 strings. This happens in Templates.pm. * In some places query needs to be uft8::decode'ed. I have done that to fix problems that showed up when Templates.pm was updated. After applying the patch, following pages need to be tested. * Patron search - Need to see all places on the results page that may contain non-latin characters. You need to have patron categories with non-latin characters. * Catalog search - Need to check places where non-latin characters appear * Non-latin templates e.g. koha in Russian language. I think this task will take a lot of time. Regards, Ketan (In reply to Bernardo Gonzalez Kriegel from comment #156)

I think that for this to pass QA and eventually been pushed we need a test plan

1) A way to check what error this patch is trying to fix, i.e. how and where to look encoding problems (add sample record/data) 2) How it's fixed

In it's current incarnation this is a simple patch, but it's not clear how to test.

-- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #159 from Ketan Kulkarni <ketan@carvingit.com> --- Hello Tomás, I appreciate your concerns. I have read the web page. It's up to the QA and RM to decide. Nevertheless, the updates in the patch have been done in accordance with the principle - decode the input and encode the output. Although utf8::upgrade() is a temporary fix to "hide" the problems if any, does fix things to a great extent. Besides, upgrade() is an inbuilt perl function. And on production systems it's better we use upgrade() than just encode() if it is going to break things. Once we have a plan in place, this will be replaced by Encode::.... You see, there's talk about using Encode::... instead of utf8::.... In the current scenario, utf8::upgrade() appears to be a decent (and forgiving) way of handling errors in coding. There's another bug related to writing a generic subroutine for utf-decoding for input. The discussion is on. Considering the size of the code-base, UTF handling related updates are going to take time. The present patch tries to fix the problem that is there in the stable release temporarily. Regards, Ketan (In reply to Tomás Cohen Arazi from comment #158)

Ketan,

(In reply to Ketan Kulkarni from comment #155)

...

Hello Tomás,

Yes, we should move from utf8::encode/decode to Encode:: This particular bug (6554) is about double encoding that happens in Templates.pm.

Take a look at this work from Marcel: http://wiki.koha-community.org/wiki/Handling_UTF8_in_development

If we (currently) do things wrong inside Koha regarding encoding/decoding of UTF-8 data (double encoding means we are encoding wrong somewhere), then fixing those symptms will just hide the problem (using utf8::upgrade will avoid the consequences but not fix the root cause, it might even hide new bugs introduced by future patches).

It is up to the QA team and the RM to think of the best approach for tackling this so I'm just leaving this comment, also I appreciate you efforts and hope this gets fixed soon.

-- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #161 from Dobrica Pavlinusic <dpavlin@rot13.org> --- Just to add my $0.02 since I was original author of encoding fixes: this patch indeed goes against idea of correctly encoding everything in Koha, but does fix some really broken places in current code. I would appreciate if this patch was accepted upstream (encoding of borrower characters is my most pressing concern) and if nothing else it does mark few more places in which we will have to take a look when making koha utf-8 clean. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #163 from Ketan Kulkarni <ketan@carvingit.com> --- Created attachment 21937 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=21937&action=edit My opac after search My opac search does not break after applying the patch. Could you please upload a screenshot? This is from the latest master. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #165 from Ketan Kulkarni <ketan@carvingit.com> --- Created attachment 21955 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=21955&action=edit My opac xslt preferences Please see my OPAC preferences. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 --- Comment #167 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 21978 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=21978&action=edit opac-search Please look at mangled characters with diacritics in the XSLT blocks and facets. -- You are receiving this mail because: You are watching all bug changes.