[Koha-bugs] [Bug 11811] tools/import_borrowers.pl doesn't support utf-8 encoded CSV

[bugzilla-daemon at bugs.koha-community.org]

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11811

Galen Charlton <gmcharlt at gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|Passed QA                   |In Discussion
                 CC|                            |gmcharlt at gmail.com

--- Comment #12 from Galen Charlton <gmcharlt at gmail.com> ---
(In reply to Katrin Fischer from comment #11)
> Use case for this is XLS file converted into CSV which includes utf-8
> characters which display correctly in interface but are not searchable
> in Koha's patrons search since browser does same decomposition as this
> code does.

Not so.  There is no guarantee that strings entered via the web browser, or
patron searches entered via the web browser, will end up in NFC.  Consequently,
one can end up with a mixture of NFD and NFC data in the database, meaning that
patron searches will not necessarily bring up the same results.

One of the root issues is the default collation that is typically used.  In my
development database, it happens to be utf8_general_ci -- which in general is
problematic [1], and in particular doesn't treat the NFD and NFC forms of
Dobrica's surname as equivalent.  Spot-checking suggests that utf8_general_ci
ends up as the default for databases installed from Debian packages, with the
exception of a couple columns that kohastructure.sql sets to utf8_unicode_ci.

Rather than piecemeal sticking in NFC(), I think a better solution would be to
change the collation to utf8_unicode_ci across the board.

[1]
http://stackoverflow.com/questions/1036454/what-are-the-diffrences-between-utf8-general-ci-and-utf8-unicode-ci

-- 
You are receiving this mail because:
You are watching all bug changes.