[Bug 21874] New: Umlauts broken in list email subjects
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21874 Bug ID: 21874 Summary: Umlauts broken in list email subjects Change sponsored?: --- Product: Koha Version: 17.11 Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Lists Assignee: koha-bugs@lists.koha-community.org Reporter: katrin.fischer@bsz-bw.de QA Contact: testopia@bugs.koha-community.org CC: m.de.rooy@rijksmuseum.nl To test: - Create a list in staff or OPAC with the title "Jugendbücher" (Juvenile books) - Send yourself the list by email - Verify that the subject is broken "Ihre Liste: Jugendbücher" As this is from the OPAC, it's quite an issue for libraries using languages with umlauts and other diacritics. I haven't been able to verify in master yet (mine can't send emails and bywater sandboxes gave me an error?), but verified it still happens in our 17.11 installations. It was inititally reported for 3.22, but got lost in a comment on bug 13245. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21874 --- Comment #1 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Will have a look tomorrow.. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21874 --- Comment #2 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- That would be great, thx Marcel! -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21874 --- Comment #3 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to Katrin Fischer from comment #0)
I haven't been able to verify in master yet (mine can't send emails and bywater sandboxes gave me an error?), but verified it still happens in our 17.11 installations. It was inititally reported for 3.22, but got lost in a comment on bug 13245.
13245 is about sending the cart, not about lists -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21874 --- Comment #4 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- I just tested a list with Jügend and sending it worked fine with 16.11 and latest master. Still looking a bit further.. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21874 --- Comment #5 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Marcel de Rooy from comment #4)
I just tested a list with Jügend and sending it worked fine with 16.11 and latest master. Still looking a bit further..
On the other bug it was also reported working by someone. I got no idea what the difference could be. Our installations are packages with Plack and memcached, no changes in the list related code. Thx for investigating! -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21874 --- Comment #6 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Non-latin works fine like perl Jügend 遺跡発見 áéíóú 遱葑癱覂. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21874 --- Comment #7 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Running some additional tests it looks like not all email clients show the display issue. It appears nicely in Thunderbird and another web based GUI, but broken in Outlook. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21874 --- Comment #8 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Hm, maybe another clue: Umlauts in the DUE and DUEDGST notices work ok for us in all email clients. They translate to: Fälligkeitsbenachrichtigung and we never had any issues there. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21874 --- Comment #9 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Trying to pinpoint the difference. The queue emails have this: 1286 my $utf8 = decode('MIME-Header', $message->{'subject'} ); 1287 $message->{subject}= encode('MIME-Header', $utf8); 1288 my $subject = encode('UTF-8', $message->{'subject'}); The shelf has: 114 # Analysing information and getting mail properties 115 if ( $template_res =~ /<SUBJECT>(.*)<END_SUBJECT>/s ) { 116 $mail{'subject'} = Encode::encode("UTF-8", $1); 117 $mail{subject} =~ s|\n?(.*)\n?|$1|; Wonder if the encode('MIME-Header'... is the magical difference? -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21874 --- Comment #10 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to Katrin Fischer from comment #9)
Trying to pinpoint the difference. The queue emails have this:
1286 my $utf8 = decode('MIME-Header', $message->{'subject'} ); 1287 $message->{subject}= encode('MIME-Header', $utf8); 1288 my $subject = encode('UTF-8', $message->{'subject'});
The shelf has:
114 # Analysing information and getting mail properties 115 if ( $template_res =~ /<SUBJECT>(.*)<END_SUBJECT>/s ) { 116 $mail{'subject'} = Encode::encode("UTF-8", $1); 117 $mail{subject} =~ s|\n?(.*)\n?|$1|;
Wonder if the encode('MIME-Header'... is the magical difference?
This is very nasty code. Minefield (mime..mine :) Lines 1286 and 1287 are obscure. An undocumented tric to decode/encode. Weird that it works. My first thought would be: $message->subject is from the database and is in perl internal format, not in MIME encoding. The decode does not result in utf8 so the var name is a bad choice(!). Should not do very much? The line 1287 encodes to MIME. Line 1288 is weird after encoding to MIME. Using still Mail::Sendmail with all its limitations in encoding etc. is another pitfall. So we should not be too hasty here. But all this encoding/decoding before calling sendmail should be moved away to one single testable module and no leftovers in several scripts. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21874 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |martin.renvoize@ptfs-europe | |.com --- Comment #11 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- For reference, there was some great discussion to narrow this one down on irc: http://irc.koha-community.org/koha/2018-11-23#i_2099148 -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21874 --- Comment #12 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Just a highlight: Your list : =?UTF-8?B?IHBlcmwgSsO8Z2VuZCDpgbrot6Hnmbroposgw6HDqcOtw7PDuiDpgbHokZHnmbE=?= =?UTF-8?B?6KaC?= perl Jügend 遺跡発見 áéíóú 遱葑癱覂 -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21874 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Version|17.11 |master Summary|Umlauts broken in list |Encoding broken in list and |email subjects |cart email subjects Assignee|koha-bugs@lists.koha-commun |katrin.fischer@bsz-bw.de |ity.org | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21874 --- Comment #13 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 82603 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=82603&action=edit Bug 21874: Fix encoding of cart and list email subjects -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21874 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21874 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #82603|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21874 --- Comment #14 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 82604 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=82604&action=edit Bug 21874: Fix encoding of cart and list email subjects -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21874 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|--- |Small patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21874 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #82604|0 |1 is obsolete| | --- Comment #15 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 82605 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=82605&action=edit Bug 21874: Fix encoding of cart and list email subjects The subject of the cart and list emails is not correctly encoded and displays incorrectly in some email clients. To test: Lists: - Create a list, name it using umlauts or other diacritics Example: Jugendbücher (books for youths in German) - Add some items to your list - Email yourself the list - Verify that the email subject is broken Cart: - Install another language with non-latin characters like Greek - Fill the cart with some titles - Send yourself the cart - Verify that the email subject is broken Note: Some email clients display correctly, others not. It's known to be incorrect in Outlook and web.de, displaying nicely in Thunderbird. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21874 --- Comment #16 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 82606 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=82606&action=edit Bug 21874: (QA follow-up) Fix intranet sendshelf Move encode line below regex line. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21874 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #82606|0 |1 is obsolete| | --- Comment #17 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 82607 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=82607&action=edit Bug 21874: (QA follow-up) Fix intranet sendshelf Move encode line below regex line. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21874 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #82607|0 |1 is obsolete| | --- Comment #18 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 82608 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=82608&action=edit Bug 21874: (QA follow-up) Fix intranet sendshelf Move encode line below regex line. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21874 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21874 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #82605|0 |1 is obsolete| | --- Comment #19 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 82609 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=82609&action=edit Bug 21874: Fix encoding of cart and list email subjects The subject of the cart and list emails is not correctly encoded and displays incorrectly in some email clients. To test: Lists: - Create a list, name it using umlauts or other diacritics Example: Jugendbücher (books for youths in German) - Add some items to your list - Email yourself the list - Verify that the email subject is broken Cart: - Install another language with non-latin characters like Greek - Fill the cart with some titles - Send yourself the cart - Verify that the email subject is broken Note: Some email clients display correctly, others not. It's known to be incorrect in Outlook and web.de, displaying nicely in Thunderbird. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Tested all four scripts by manipulating the cart text too in templates. Follow-up handles intranet sendshelf. Note: I asked Katrin to remove the encode UTF-8 statements, since we convert to MIME and the subject line should never be UTF-8. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21874 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #82608|0 |1 is obsolete| | --- Comment #20 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 82610 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=82610&action=edit Bug 21874: (QA follow-up) Fix intranet sendshelf Move encode line below regex line. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21874 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |martin.renvoize@ptfs-europe |y.org |.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21874 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |rel_18_11_candidate -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21874 --- Comment #21 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to Marcel de Rooy from comment #19)
Note: I asked Katrin to remove the encode UTF-8 statements, since we convert to MIME and the subject line should never be UTF-8.
Hope that was clear. No UTF-8 in the header lines. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21874 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #82609|0 |1 is obsolete| | --- Comment #22 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 82612 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=82612&action=edit Bug 21874: Fix encoding of cart and list email subjects The subject of the cart and list emails is not correctly encoded and displays incorrectly in some email clients. To test: Lists: - Create a list, name it using umlauts or other diacritics Example: Jugendbücher (books for youths in German) - Add some items to your list - Email yourself the list - Verify that the email subject is broken Cart: - Install another language with non-latin characters like Greek - Fill the cart with some titles - Send yourself the cart - Verify that the email subject is broken Note: Some email clients display correctly, others not. It's known to be incorrect in Outlook and web.de, displaying nicely in Thunderbird. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Tested all four scripts by manipulating the cart text too in templates. Follow-up handles intranet sendshelf. Note: I asked Katrin to remove the encode UTF-8 statements, since we convert to MIME and the subject line should never be UTF-8. Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21874 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #82610|0 |1 is obsolete| | --- Comment #23 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 82613 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=82613&action=edit Bug 21874: (QA follow-up) Fix intranet sendshelf Move encode line below regex line. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21874 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA --- Comment #24 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- This fix had lots of eyes on it :) No tests required as it's all in scripts and not modules.. however, we really should look at centralising this code sooner rather than later, but that effort should not hold off this bugfix. Passing QA. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21874 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nick@bywatersolutions.com Status|Passed QA |Pushed to Master --- Comment #25 from Nick Clemens <nick@bywatersolutions.com> --- Awesome work all! Pushed to master for 18.11 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21874 Jesse Maseto <jesse@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to Master |Pushed to Stable CC| |jesse@bywatersolutions.com --- Comment #26 from Jesse Maseto <jesse@bywatersolutions.com> --- Pushed to stable for 18.05.07 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21874 Fridolin SOMERS <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m --- Comment #27 from Fridolin SOMERS <fridolin.somers@biblibre.com> --- Pushed to 17.11.x for 17.11.13 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21874 Fridolin SOMERS <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED Status|Pushed to Stable |RESOLVED -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org