https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17359 --- Comment #1 from Mirko Tietgen <mirko@abunchofthings.net> --- Card numbers can be non numeric, use a too long one instead. (Limit is 16 I think) -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17359 Anke <anke.bruns@gwdg.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |anke.bruns@gwdg.de --- Comment #3 from Anke <anke.bruns@gwdg.de> --- As I noticed, the German Umlauts aren't displayed correctly after patron import even in patron names, patron attributes and other fields. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17359 --- Comment #5 from Anke <anke.bruns@gwdg.de> --- (In reply to Anke from comment #4)

(In reply to Anke from comment #3)

...

As I noticed, the German Umlauts aren't displayed correctly after patron import even in patron names, patron attributes and other fields.

Even with UTF 8 chosen for encoding, as mentioned in the documentation, the umlaute are shown correctly in the import "log" list, but in the "real" patron entries they are still shown as question marks. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17359 --- Comment #7 from Anke <anke.bruns@gwdg.de> --- (In reply to Katrin Fischer from comment #6)

Hi Anke, could you share a broken sample file here (Add an attachment)? Which version are you testing on?

Dear Katrin, because of the personal data it seems a bit delicate to send a sample; with some more time I can try to import some fantasy patrons and send them. Some more infos for now: - The problem affects not only German umlauts but also accents. Thus, Timón becomes Tim?n in Koha. - It is not limited to the patrons' names but also to self-defined attributes (<koha-url>/cgi-bin/koha/admin/patron-attr-types.pl); e.g.: I defined an attribute "Gruppe" which, for some users has a name with an umlaut as content, e.g. Gruppe: Müller. This shows as Gruppe: M?ller I thought there might be something wrong with my browser's encoding, but even when I changed this from UTF8 to "Westeuropäisch", the question marks instead of umlauts and accents remain. I also tried a second browser (IE) to make sure my Firefox doesn't cause the problem. My Koha version is 19.05.02.000. I got the data from a csv file which I edited with Notepad++, set the encoding to "UTF8 ohne BOM" (just UTF8 will result in an error message when loading the patrons into Koha). Hope this helps to identify the cause for the bug. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17359 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|normal |major -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17359 --- Comment #10 from Anke <anke.bruns@gwdg.de> --- OK, I imported some fantasy users into my Koha; I used a comma separated .csv file See attached .csv as well as a screenshot of the import summary (where the umlauts and accents look good) and two more screenshots of the list of patrons and a detailed patron information showing that here, the umlauts and accents aren't displayed correctly. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17359 --- Comment #12 from Anke <anke.bruns@gwdg.de> --- Created attachment 93928 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=93928&action=edit Patron list -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17359 Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |caroline.cyr-la-rose@inlibr | |o.com --- Comment #14 from Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com> --- I was able to reproduce the problem mentionned by Mirko, i.e. that the error summary has encoding problems. The user's name was Chloé and it came out as Chloé BUT once imported, that user's file was fine. So I can't reproduce Anke's problem. For reference, I'm including a one-line file with the name and other information changed, but I kept the same firstname. You can try to import it as is and the error will be that the branchcode and categorycodes don't exist. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17359 --- Comment #16 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I downloaded Anke's sample file, but my editor (Geany) and LibreOffice both recognize it as "Western ISO-8859-1". So I think something might have gone wrong with encoding conversion, hoping Bugzilla is not to blame here. Could you please try this: - Open the file in Notepad++ - Check the setting it displays under 'Encoding' - If it displays as something other than ANSI, change it to ANSI - Use Encoding > Convert to UTF-8 (without BOM) in the lower half of the menu - Verify if your Umlauts look correct and that the encoding setting displays right as UTF-8 - Save I could only check the menu options in the Notepad++ Linux version, so they might be named slightly different in the German/Windows version. Caroline's file behaves well and displays as UTF-8 in all my programs. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17359 --- Comment #18 from Anke <anke.bruns@gwdg.de> --- Thanks to Katrin and Caroline and all! It was really a question of the encoding I tried to do with Notepad++, so no problem in Koha; encoding as described by Katrin (post of 2019-10-09 20:43:52 UTC) worked, and I apologize for bothering everybody! Sorry indeed! -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17359 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17359 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|REOPENED |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17359 --- Comment #22 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 94262 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=94262&action=edit Bug 17359: Correct encoding when displaying patron import summary There is an encoding issue when the patron import summry, the string must be UTF8 decoded before being sent to the template. Test plan: - Create a patron import CSV file with import issues. Use UTF8 characters. See file attached to the bug report 'patron_import.csv' => The screen should display the line correctly - Correct the import issue (add a surname for Chloé) => The imported patrons should contain the correct values. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17359 Séverine Queune <severine.queune@bulac.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17359 Séverine Queune <severine.queune@bulac.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #94262|0 |1 is obsolete| | --- Comment #25 from Séverine Queune <severine.queune@bulac.fr> --- Created attachment 94810 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=94810&action=edit Bug 17359: Correct encoding when displaying patron import summary There is an encoding issue when the patron import summry, the string must be UTF8 decoded before being sent to the template. Test plan: - Create a patron import CSV file with import issues. Use UTF8 characters. See file attached to the bug report 'patron_import.csv' => The screen should display the line correctly - Correct the import issue (add a surname for Chloé) => The imported patrons should contain the correct values. Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17359 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|BLOCKED |Failed QA --- Comment #27 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- QA Comment: my $surname ='Chloé'; my $surname_utf8 = encode_utf8($surname); => This name is not the best example for UTF-8. Could be Latin too. => You dont do anything with this encoded string. => When I am adding some Chinese character in the surname, the test fails. decode_utf8($borrowerline) Instead of adding this all over the place, we should try to fix at the source or at least closer. LINE: while ( my $borrowerline = <$handle> ) { You could add it here, which would be closer but thats not all. Is the assumption correct that it is UTF-8? We just got a handle from the caller. On the report I am reading about people uploading Latin data. Actually, we see in tools/import_borrowers.pl: use CGI qw ( -utf8 ); and my $handle = $input->upload('uploadborrowers'); CGI tells me that "binary upload data ... are left untouched". Do we actually need another parameter for the file encoding? Looks like this needs some more attention. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17359 --- Comment #28 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 95042 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=95042&action=edit Bug 17359: Use a unicode character in tests And remove an unused variable. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17359 --- Comment #30 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to Jonathan Druart from comment #28)

Created attachment 95042 [details] [review] Bug 17359: Use a unicode character in tests

And remove an unused variable.

t/db_dependent/Koha/Patrons/Import.t | 2 +- Where did you remove it? :) -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17359 --- Comment #32 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to Jonathan Druart from comment #29)

As it seems that there is no encoding issue during the import process I tried to provide a patch that will not have impact on it. That's why I preferred to only modify the lines where @missing_criticals was used.

I think that Text::CSV resolves the issue for us by guessing the encoding correctly. Formally, we cannot even pass it the encoding because we don't know. So I can understand your point. And will pass QA. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17359 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #94809|0 |1 is obsolete| | --- Comment #33 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 95057 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=95057&action=edit Bug 17359: Add tests Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17359 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #95042|0 |1 is obsolete| | --- Comment #35 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 95059 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=95059&action=edit Bug 17359: Use a unicode character in tests And remove an unused variable. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> [EDIT] Now really removed it. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17359 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |19.11.00 released in| | Status|Passed QA |Pushed to master -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17359 Fridolin SOMERS <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m Status|Pushed to master |Pushed to stable Version(s)|19.11.00 |19.11.00,19.05.05 released in| | --- Comment #38 from Fridolin SOMERS <fridolin.somers@biblibre.com> --- Pushed to 19.05.x for 19.05.05 -- You are receiving this mail because: You are watching all bug changes.