http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8872 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|enhancement |minor --- Comment #1 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- I had some incidental cases where diacritical characters a.o. were wrongly encoded in Koha. This appeared to originate from the Template output routine. This routine encodes all variables before sending them out. But merging these values with other non-encoded variables [somewhere in the template process call] may occasionally let Perl make a wrong choice when encoding/decoding. The attached patches are proposed to resolve the situation and make Koha more consistent. Patch 1 Adds specific encoding parameter when creating Template object. Note that the instruction binmode => ':utf8' could have been added also to the process method. But since it outputs to a Perl variable, this is not needed. Finally, instead of the old encoding of all parameters before processing, the output of process is encoded to UTF-8 octets. Note that Perl could have handled this internally, but this would require adding a use open statement, setting IO layers to UTF-8 in all scripts. Patch 2 Add UTF-8 encoding to intranet xslt files in output clause. Patch 3 Add UTF-8 encoding to opac xslt files also. Does also fix head and body tags in MARC21slim2OPACMARCdetail.xsl. Patch 4 Simplifies opac-showmarc template (eliminating opac-bottom include). Makes encoding for card and html view more consistent with approach in Templates module. Rearranges a few lines in script for consistency and performance. Patch 5 Simplifies showmarc template (eliminating intranet-bottom include). Makes encoding for card view more consistent with approach in Templates module. Rearranges a few lines in script for consistency and performance. All patches should be combined. Finding a test case may be quite hard. Please choose some records with special characters. And check in several opac places and in staff that they are displayed correctly. Especially check MARC view/plain MARC view in opac and MARC/Card view in intranet results. Also check normal view in XSLT mode and non-XSLT mode. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8872 --- Comment #3 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 12756 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=12756&action=edit Patch 2 for intranet xslt -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8872 --- Comment #5 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 12758 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=12758&action=edit Patch 4 for opac-showmarc -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8872 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8872 Jared Camins-Esakov <jcamins@cpbibliography.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #12755|0 |1 is obsolete| | Attachment #12756|0 |1 is obsolete| | Attachment #12757|0 |1 is obsolete| | Attachment #12758|0 |1 is obsolete| | Attachment #12759|0 |1 is obsolete| | --- Comment #7 from Jared Camins-Esakov <jcamins@cpbibliography.com> --- Created attachment 12986 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=12986&action=edit Bug 8872 Encoding issues in Templates.pm Adds specific encoding parameter when creating Template object. Note that the instruction binmode => ':utf8' could have been added also to the process method. But since it outputs to a Perl variable, this is not needed. Finally, instead of the old encoding of all parameters before processing, the output of process is encoded to UTF-8 octets. Note that Perl could have handled this internally, but this would require adding a use open statement, setting IO layers to UTF-8 in all scripts. Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com> Successfully tested on a record that displayed the UTF-8 "unknown character" glyph before the patch, at least in plain MARC, and the correct character after. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8872 --- Comment #9 from Jared Camins-Esakov <jcamins@cpbibliography.com> --- Created attachment 12988 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=12988&action=edit Bug 8872 Add UTF-8 encoding to opac xslt files Does also fix head and body tags in MARC21slim2OPACMARCdetail.xsl. Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com> -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8872 --- Comment #11 from Jared Camins-Esakov <jcamins@cpbibliography.com> --- Created attachment 12990 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=12990&action=edit Bug 8872: Changes for intranet showmarc Simplifies template (eliminating intranet-bottom include). Makes encoding for card view more consistent with approach in Templates module. Rearranges a few lines in script for consistency and performance. Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com> -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8872 --- Comment #13 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to comment #12)

I'm a little bit reluctant to push this patc for 3.10 because it changes things in the core of Koha. If we miss something, it can have very bad side effect. Do people here agree to wait for after 3.10.0 release ?

Well, I do not mind holding this patch a little longer. Probably other patches touching core modules should wait now too. About the very bad side effect: We use this in production for some time already. Note that it should still be QAed. Did you do that? -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8872 Mason James <mtj@kohaaloha.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #13171|0 |1 is obsolete| | --- Comment #15 from Mason James <mtj@kohaaloha.com> --- Created attachment 13172 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=13172&action=edit Bug 8872: Changes for intranet showmarc Simplifies template (eliminating intranet-bottom include). Makes encoding for card view more consistent with approach in Templates module. Rearranges a few lines in script for consistency and performance. Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com> -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8872 --- Comment #17 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Thanks for rebasing! But I discovered now that this patch needs a small followup for opac facets and opac search results paging. Somehow diacritics are not handled consistently there yet.. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8872 --- Comment #19 from Mason James <mtj@kohaaloha.com> --- (In reply to comment #18)

Created attachment 13173 [details] Subject: [PATCH] Bug 8872: Changes for opac-showmarc

Simplifies template (eliminating opac-bottom include). Makes encoding for card and html view more consistent with approach in Templates module. Rearranges a few lines in script for consistency and performance.

Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>

meh, and a rebase for this one too... -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8872 --- Comment #21 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Please go ahead. A followup will need additional signoff ;) -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8872 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #13173|0 |1 is obsolete| | --- Comment #23 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 13244 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=13244&action=edit Bug 8872: Changes for opac-showmarc -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8872 --- Comment #25 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- I have removed the patch for Templates.pm since this would require some more changes to Koha than planned for this report. Koha should follow the principle: decode everything coming in from outside, encode everything when sending it to outside. But in several places this does not happen: especially for url parameters and template file contents. The first patch did solve the template file contents but since it would not touch all places where url parameters are passed in, it is not fully working. What remains in this report, is a fix for xslt files and opac-showmarc and showmarc. I think that in this form the patches could be pushed (after QA). I will ask Jared to confirm since he signed off on the full set of five patches. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8872 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #13172|0 |1 is obsolete| | --- Comment #26 from M. de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 13245 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=13245&action=edit Bug 8872: Changes for intranet showmarc -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8872 --- Comment #28 from Jared Camins-Esakov <jcamins@cpbibliography.com> --- (In reply to comment #25)

I have removed the patch for Templates.pm since this would require some more changes to Koha than planned for this report. Koha should follow the principle: decode everything coming in from outside, encode everything when sending it to outside. But in several places this does not happen: especially for url parameters and template file contents. The first patch did solve the template file contents but since it would not touch all places where url parameters are passed in, it is not fully working.

What remains in this report, is a fix for xslt files and opac-showmarc and showmarc. I think that in this form the patches could be pushed (after QA).

I will ask Jared to confirm since he signed off on the full set of five patches.

Agreed. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8872 Paul Poulain <paul.poulain@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version|master |3.10 --- Comment #31 from Paul Poulain <paul.poulain@biblibre.com> --- Patch pushed to branch 3.10.x (I much more prefer those ones than the previous ones -from side-effect risk POV- !) -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8872 Liz Rea (CatalystIT) <liz@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to Stable |RESOLVED CC| |liz@catalyst.net.nz Resolution|--- |FIXED --- Comment #33 from Liz Rea (CatalystIT) <liz@catalyst.net.nz> --- Patches do not apply to 3.6.x, please reopen and submit a patch for 3.6.x if necessary. Thanks! -- You are receiving this mail because: You are watching all bug changes.