https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17842 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=17754 -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17842 --- Comment #2 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Hm, no. I get this when I export using the MARC option from the download menu. The attachement to the email (send cart) is fine. Which encoding did you pick for importing? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17842 --- Comment #4 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I tested this on master before filing, but will test again and try to attach the exported files. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17842 --- Comment #6 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Sorry not yet! -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17842 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution|--- |WORKSFORME --- Comment #8 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I can no longer reproduce this issue on master (tested in kohadevbox). -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17842 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Broken diacritics on |Broken diacritics on |records exported as MARC |records exported as MARC |from cart on reimport |from cart -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17842 --- Comment #11 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- % marcdump ~/Downloads/cart.iso2709|grep 245 /home/jonathan/Downloads/cart.iso2709 245 10 _aäüö ❤ ✓ -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17842 --- Comment #13 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Could you attach the record here and tell us the different options you chose on importing it? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17842 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Target Milestone|Oldversions |--- Owen Leonard <oleonard@myacpl.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|oleonard@myacpl.org |koha-bugs@lists.koha-commun | |ity.org --- Comment #15 from Owen Leonard <oleonard@myacpl.org> --- I cannot recreate this problem in master. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17842 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |UNCONFIRMED Ever confirmed|1 |0 Resolution|WORKSFORME |--- --- Comment #17 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I'd really like to get behind this issue. I just retested on our Koha 18.11.16 and we still have the issue. :( Our test installation is publicly accessible, but I don't want to post the link here. Joubu, maybe I could talk you through it? For matching I just seleted the file and changed none of the other settings (encoding defaults to UTF-8). We do have one local change on that file as we did chose not to export items: my $record = GetMarcBiblio({ biblionumber => $biblio, - embed_items => 1, + embed_items => 0, But that should not make a difference in encoding? Attaching sample records. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17842 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au --- Comment #19 from David Cook <dcook@prosentient.com.au> --- Given my bad experience the other day trying to import records converted from GB2312 to UTF8 into Koha, I'm extra interested by this. Maybe it's a related topic. At a glance, those sample records look fine both in Latin1 and UTF8. MarcEdit can convert the ISO MARC into its MRK format, but I'm failing to convert it from ISO MARC to MARCXML. When I try to read your sample records as UTF-8 using MARC::File::USMARC, I see the following error: UTF-8 "\xFC" does not map to Unicode Using "xxd cart.iso2709", I see that the "fc" byte is the ü in über and für. Ah, and FC is ü in Latin-1 encoding whereas in UTF-8 it's C3 BC. So it sounds like Koha is exporting as Latin-1 but trying to import as UTF-8 and that's where it's falling over? Needs more investigating, but that's the problem with your sample records I'd say. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17842 --- Comment #20 from David Cook <dcook@prosentient.com.au> --- Oh, Katrin, try importing it using /cgi-bin/koha/tools/stage-marc-import.pl with a Character encoding of "ISO 8859-1". I bet it'll work. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17842 --- Comment #22 from David Cook <dcook@prosentient.com.au> --- (In reply to Katrin Fischer from comment #21)

Hi David, it looks like you are correct. Using ISO 8859-1 encoding when staging the MARC imports correctly. So... why do we not export in UTF-8 for the cart? I think we do in other places as for example the MARC attached to the cart email works correctly.

That's a very good question. Which Koha version are you using? This is the "Download" option in the cart? (OPAC or Staff Client?) I'll take a quick look... -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17842 --- Comment #24 from David Cook <dcook@prosentient.com.au> --- (In reply to David Cook from comment #23)

Ok it just looks like a bug to me.

opac/opac-downloadcart.pl doesn't instruct Perl how to output the bytes, and I think Perl defaults to N8CS/Latin-1.

With opac/opac-sendbasket.pl, we explicitly encode the characters as UTF8 bytes.

That being said... I can't reproduce the problem. In 19.11, I added some Chinese to a test record, and I was able to download it from Staff Client and OPAC carts and Notepad++ says they're both UTF-8 and renders correctly... I'll try with ü specifically... -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17842 --- Comment #26 from David Cook <dcook@prosentient.com.au> --- (In reply to David Cook from comment #25)

Ha! If I use ü instead of 我, it outputs as Latin-1 and ü is a FC byte instead of the C3 BC bytes.

Using 我, I get the correct bytes: e6 88 91. Now if I add ü after 我, I am guessing that whatever automatic character encoding is happening (perhaps on the browser end?) will encode ü as UTF-8 instead of Latin-1 -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17842 --- Comment #28 from David Cook <dcook@prosentient.com.au> --- Plot twist: encoding the character output as UTF-8 doesn't seem to make a difference. At least not when using Chrome, which I'm guessing is automatically transcoding the output. I'm going to see what wget does... -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17842 --- Comment #30 from David Cook <dcook@prosentient.com.au> --- Using the following doesn't help either: Content-Type: application/octet-stream; charset=utf-8 I notice the console is saying "Resource interpreted as Document but transferred with MIME type application/octet-stream", so I think that's what's doing it. More to come... -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17842 --- Comment #32 from David Cook <dcook@prosentient.com.au> --- Created attachment 105341 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=105341&action=edit Bug 17842: UTF-8 encode ISO2709 MARC download from cart The cart was outputing ISO2709 MARC records with Latin-1 encoding. Records containing non-latin1 characters were automatically re-encoded as UTF-8 browsers, which led to inconsistent character encodings for downloaded MARC files. This patch explicitly encodes ISO2709 MARC characters from the cart download as UTF-8 encoded bytes, which resolves the problem. Test Plan: 0) Don't apply patch 1) Create bib record with only ASCII characters 2) Add a ü character to the title 3) Save bib record 4) Download bib record from cart (opac and staff client) 5) Using xxd or some other program, note that the ü is represented by a FC byte (latin-1 encoded) 6) Apply the patch 7) Download bib record from cart (opac and staff client) 8) Using xxd or some other program, note that the ü is represented by C3 BC bytes (utf-8 encoded) 9) Success (Note that you could potentially use Notepad++ or some other program to open the downloaded file and just note the encoding that it finds. You could also try "chardetect" instead. Lots of options for figuring out the encoding.) -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17842 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Ever confirmed|0 |1 Status|UNCONFIRMED |NEW -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17842 --- Comment #34 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Thx David, and that finally also explains why Jonathan could never reproduce the issue - he had added other characters that were outside of latin-1. It was really bugging me that I seemed to be the only one being able to see this 'ghost bug'. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17842 --- Comment #36 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 105344 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=105344&action=edit New export file -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17842 --- Comment #38 from David Cook <dcook@prosentient.com.au> --- So interesting. It looks like the UTF-8 encoded data has been re-encoded as Latin-1. I'm guessing this is because of the browser, and I have an idea for something that might help, although I want to try to reproduce your issue first... -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17842 --- Comment #40 from David Cook <dcook@prosentient.com.au> --- (In reply to David Cook from comment #39)

I just uploaded "Sample records correctly encoded" into my 19.11 system (with the patch), and I was able to export it as UTF-8 (according to Notepad++ and the hex inspected using xxd).

So I can't reproduce the double-encoding issue.

I did my test in Chrome 81 on Windows 10.

About to test on koha-testing-docker with the same test plan.

That was also using CGI rather than Plack. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17842 --- Comment #42 from David Cook <dcook@prosentient.com.au> --- Very silly coding error on my part, which would only happen if you were exporting more than 1 record. I'll have a revised patch in a minute. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17842 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #105341|0 |1 is obsolete| | --- Comment #43 from David Cook <dcook@prosentient.com.au> --- Created attachment 105346 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=105346&action=edit Bug 17842: UTF-8 encode ISO2709 MARC download from cart The cart was outputing ISO2709 MARC records with Latin-1 encoding. Records containing non-latin1 characters were automatically re-encoded as UTF-8 by browsers, which led to inconsistent character encodings for downloaded MARC files. This patch explicitly encodes ISO2709 MARC characters from the cart download as UTF-8 encoded bytes, which resolves the problem. Test Plan: 0) Don't apply patch 1) Create bib record with only ASCII characters 2) Add a ü character to the title 3) Save bib record 4) Download bib record from cart (opac and staff client) 5) Using xxd or some other program, note that the ü is represented by a FC byte (latin-1 encoded) 6) Apply the patch 7) Download bib record from cart (opac and staff client) 8) Using xxd or some other program, note that the ü is represented by C3 BC bytes (utf-8 encoded) 9) Success (Note that you could potentially use Notepad++ or some other program to open the downloaded file and just note the encoding that it finds. You could also try "chardetect" instead. Lots of options for figuring out the encoding.) -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17842 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |victor@tuxayo.net Status|Needs Signoff |Signed Off --- Comment #45 from Victor Grousset/tuxayo <victor@tuxayo.net> --- It works! :D -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17842 --- Comment #47 from David Cook <dcook@prosentient.com.au> --- (In reply to Jonathan Druart from comment #46)

I think the whole block could be:

$output .= encode("UTF-8", $record->as_usmarc()) // q{};

From Encode::encode POD: "If the $string is undef, then undef is returned."

Personally, I don't like putting method output directly into another function call, as it can make exception handling harder. But your suggestion does look more elegant and would be less code to maintain. I'd be happy with either. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17842 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #105440|0 |1 is obsolete| | --- Comment #49 from Julian Maurice <julian.maurice@biblibre.com> --- Created attachment 105568 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=105568&action=edit Bug 17842: UTF-8 encode ISO2709 MARC download from cart The cart was outputing ISO2709 MARC records with Latin-1 encoding. Records containing non-latin1 characters were automatically re-encoded as UTF-8 by browsers, which led to inconsistent character encodings for downloaded MARC files. This patch explicitly encodes ISO2709 MARC characters from the cart download as UTF-8 encoded bytes, which resolves the problem. Test Plan: 0) Don't apply patch 1) Create bib record with only ASCII characters 2) Add a ü character to the title 3) Save bib record 4) Download bib record from cart (opac and staff client) 5) Using xxd or some other program, note that the ü is represented by a FC byte (latin-1 encoded) 6) Apply the patch 7) Download bib record from cart (opac and staff client) 8) Using xxd or some other program, note that the ü is represented by C3 BC bytes (utf-8 encoded) 9) Success (Note that you could potentially use Notepad++ or some other program to open the downloaded file and just note the encoding that it finds. You could also try "chardetect" instead. Lots of options for figuring out the encoding.) Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> Signed-off-by: Julian Maurice <julian.maurice@biblibre.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17842 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA CC| |julian.maurice@biblibre.com --- Comment #51 from Julian Maurice <julian.maurice@biblibre.com> --- (In reply to David Cook from comment #47)

(In reply to Jonathan Druart from comment #46)

...

I think the whole block could be:

$output .= encode("UTF-8", $record->as_usmarc()) // q{};

From Encode::encode POD: "If the $string is undef, then undef is returned."

Personally, I don't like putting method output directly into another function call, as it can make exception handling harder.

I don't like it either, but it should be harmless in this particular case. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17842 --- Comment #52 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Pushed to master for 20.11, thanks to everybody involved! -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17842 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |aleisha@catalyst.net.nz Version(s)|20.11.00, 20.05.01 |20.11.00, 20.05.01, released in| |19.11.07 Status|Pushed to stable |Pushed to oldstable --- Comment #54 from Aleisha Amohia <aleisha@catalyst.net.nz> --- backported to 19.11.x for 19.11.07 -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17842 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=11997 -- You are receiving this mail because: You are watching all bug changes.