[Bug 24330] New: When importing patrons from CSV, automatically strip BOM from file if it exists
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24330 Bug ID: 24330 Summary: When importing patrons from CSV, automatically strip BOM from file if it exists Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Tools Assignee: koha-bugs@lists.koha-community.org Reporter: kyle@bywatersolutions.com QA Contact: testopia@bugs.koha-community.org We have a partner that exports UTF-8 CSV files, and is experiencing the same thing as the author of this article: https://www.freecodecamp.org/news/a-quick-tale-about-feff-the-invisible-char... In short, Excel is inserting an invisible UTF-8 character at the start of the file, so that the column name "cardnumber" is actually named "\x{feff}cardnumber", causing "cardnumber" to be blank. A simple solution is provided here: https://stackoverflow.com/questions/24390034/remove-bom-from-string-with-per... -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24330 --- Comment #1 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 96762 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=96762&action=edit Example CSV with a BOM -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24330 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24330 --- Comment #2 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 96763 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=96763&action=edit Bug 24330: When importing patrons from CSV, automatically strip BOM from file if it exists We have a partner that exports UTF-8 CSV files, and is experiencing the same thing as the author of this article: https://www.freecodecamp.org/news/a-quick-tale-about-feff-the-invisible-char... In short, Excel is inserting an invisible UTF-8 character at the start of the file, so that the column name "cardnumber" is actually named "\x{feff}cardnumber", causing "cardnumber" to be blank. A simple solution is provided here: https://stackoverflow.com/questions/24390034/remove-bom-from-string-with-per... Test Plan: 1) Download the example.csv file 2) Attempt to import it using the patron import tool 3) Note the invalid column name error 4) Apply this patch, restart all the things! 5) Attempt the import again 6) Assuming you have a branchcode MPL and a cataegory code S, the patron should import! -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24330 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |kyle@bywatersolutions.com |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24330 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24330 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #96763|0 |1 is obsolete| | --- Comment #3 from David Nind <david@davidnind.com> --- Created attachment 96789 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=96789&action=edit Bug 24330: When importing patrons from CSV, automatically strip BOM from file if it exists We have a partner that exports UTF-8 CSV files, and is experiencing the same thing as the author of this article: https://www.freecodecamp.org/news/a-quick-tale-about-feff-the-invisible-char... In short, Excel is inserting an invisible UTF-8 character at the start of the file, so that the column name "cardnumber" is actually named "\x{feff}cardnumber", causing "cardnumber" to be blank. A simple solution is provided here: https://stackoverflow.com/questions/24390034/remove-bom-from-string-with-per... Test Plan: 1) Download the example.csv file 2) Attempt to import it using the patron import tool 3) Note the invalid column name error 4) Apply this patch, restart all the things! 5) Attempt the import again 6) Assuming you have a branchcode MPL and a cataegory code S, the patron should import! Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24330 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@bugs.koha-c | |ommunity.org --- Comment #4 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- The example CSV file works correctly for me without the patch. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24330 --- Comment #5 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Do you get an error in the log? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24330 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #96789|0 |1 is obsolete| | --- Comment #6 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 98156 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=98156&action=edit Bug 24330: When importing patrons from CSV, automatically strip BOM from file if it exists We have a partner that exports UTF-8 CSV files, and is experiencing the same thing as the author of this article: https://www.freecodecamp.org/news/a-quick-tale-about-feff-the-invisible-char... In short, Excel is inserting an invisible UTF-8 character at the start of the file, so that the column name "cardnumber" is actually named "\x{feff}cardnumber", causing "cardnumber" to be blank. A simple solution is provided here: https://stackoverflow.com/questions/24390034/remove-bom-from-string-with-per... Test Plan: 1) Download the example.csv file 2) Attempt to import it using the patron import tool 3) Note the invalid column name error 4) Apply this patch, restart all the things! 5) Attempt the import again 6) Assuming you have a branchcode MPL and a cataegory code S, the patron should import! Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Yew Chung College Library <library@yccece.edu.hk> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24330 --- Comment #7 from Kyle M Hall <kyle@bywatersolutions.com> --- (In reply to Jonathan Druart from comment #4)
The example CSV file works correctly for me without the patch.
I must admit I can no longer recreate the error, it appears I've lost the specific file that caused the error. The uploaded file no longer has the feff bom. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24330 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #98156|0 |1 is obsolete| | --- Comment #8 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 98173 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=98173&action=edit Bug 24330: When importing patrons from CSV, automatically strip BOM from file if it exists We have a partner that exports UTF-8 CSV files, and is experiencing the same thing as the author of this article: https://www.freecodecamp.org/news/a-quick-tale-about-feff-the-invisible-char... In short, Excel is inserting an invisible UTF-8 character at the start of the file, so that the column name "cardnumber" is actually named "\x{feff}cardnumber", causing "cardnumber" to be blank. A simple solution is provided here: https://stackoverflow.com/questions/24390034/remove-bom-from-string-with-per... Test Plan: 1) Download the example.csv file 2) Attempt to import it using the patron import tool 3) Note the invalid column name error 4) Apply this patch, restart all the things! 5) Attempt the import again 6) Assuming you have a branchcode MPL and a cataegory code S, the patron should import! Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Yew Chung College Library <library@yccece.edu.hk> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24330 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |martin.renvoize@ptfs-europe | |.com Status|Signed Off |Passed QA --- Comment #9 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- I've run up against this one a few times myself.. I always wondered if we should be reading the BOM and doing something with it.. but actually I could never find any documentation as to 'what' to do.. so.. removing it makes sense to me. Passing QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24330 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |20.05.00 released in| | Status|Passed QA |Pushed to master -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24330 --- Comment #10 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Nice work everyone! Pushed to master for 20.05 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24330 Joy Nelson <joy@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |joy@bywatersolutions.com Status|Pushed to master |Pushed to stable Version(s)|20.05.00 |20.05.00, 19.11.03 released in| | --- Comment #11 from Joy Nelson <joy@bywatersolutions.com> --- Pushed to 19.11.x branch for 19.11.03 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24330 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lucas@bywatersolutions.com Status|Pushed to stable |Pushed to oldstable Version(s)|20.05.00, 19.11.03 |20.05.00, 19.11.03, released in| |19.05.08 --- Comment #12 from Lucas Gass <lucas@bywatersolutions.com> --- backported to 19.05.x for 19.05.08 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24330 Hayley Mapley <hayleymapley@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |hayleymapley@catalyst.net.n | |z Version(s)|20.05.00, 19.11.03, |20.05.00, 19.11.03, released in|19.05.08 |19.05.08, 18.11.14 Status|Pushed to oldstable |Pushed to oldoldstable --- Comment #13 from Hayley Mapley <hayleymapley@catalyst.net.nz> --- Backported to 18.11.x for 18.11.14. -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org