[Koha-devel] binmode in pler scripts for utf-8 support

[Rick Welykochy]

Sébastien Nadeau wrote:

 > " To mark FILEHANDLE as UTF-8, use :utf8  or :encoding(utf8) . :utf8  just marks the data as UTF-8 without further checking, while :encoding(utf8)  checks the data for actually being valid UTF-8. More details can be found in PerlIO::encoding. "
 > This quote from the binmode perldoc suggests that one must take good care about what is written to the filehandle when the ":uft8" layer is used. It it's not well formed uft-8, the result might end up in something unexpected.


Joe Atzberger wrote:

> That is just one form that happens to be used for existing filehandles.  
> You can also find similar code in "open" statements (see 
> rebuild_zebra.pl) and the same code in different forms like:
> binmode(STDOUT,":utf8");
> 
> We should setting it (or something like it) where needed. 

Exactly where is it needed?

Is it required for web page output to stdout?
Is it required when Koha creates a CSV file as output?

Is the situation as it stands a bug?

 From what M.Nadeau indicates, we should be using :encoding(utf8),
which ensures that our UTF-8 output is valid. It is another form
of turning on warnings. ":utf8" is not enough.

The problem with UTF-8 and non-Latin character sets in general is
that most of the developers never venture into accented character
sets and thus support for UTF-8 has been late in coming to Koha.

We are lucky to have many non-English language contributors who
can keep the project "honest" about character encoding.

I must admit that I myself do not test or develop any software
using UTF-8, unless perhaps it is turned on by default. I certainly
would not be aware if non-English alphabets were being broken by
anything I did to the software.

This concerns me and I would like to ensure we do as much as possible
in the Koha software to ensure we get it right.


cheers
rickw



-- 
_________________________________
Rick Welykochy || Praxis Services

It isn't pollution that's harming the environment. It's the impurities in our air
and water that are doing it.
      -- Al Gore, Vice President