- Koha-devel - lists.koha-community.org

BIG BIG problem (utf8 & mySQL)
by Paul POULAIN 17 Feb '06

17 Feb '06

Hello, (I cc: mike Rylander in case the pines guys have another solution...) I continue investigating the utf8 stuff. I have a very very bad feeling : it seems that Perl DBI::mysql does not handle utf8 correctly. Here are some links : http://www.cpanforum.com/threads/654 http://lists.mysql.com/perl/3714 http://marc.theaimsgroup.com/?l=msql-mysql-modules&m=111970179409036&w=2 http://lists.mysql.com/perl/3006?f=plain If I read correctly, there are only 2 ways to solve this : * bug DBD::mysql maintainer to have the fix included. Would require an official release, an upgrade of DBD::mysql, but it's a better solution in the long term. * modify EVERY SQL query to Encode::decode_utf8() every value. A quite huge task ! I have modified one script (marc_subfield_structure.pl), and everything works fine with Encode::decode_utf8 (and don't work without it) -- Paul POULAIN et Henri Damien LAURENT Consultants indépendants en logiciels libres et bibliothéconomie (http://www.koha-fr.org)

2 2

cvs head & templates
by Paul POULAIN 17 Feb '06

17 Feb '06

Hello all, i've seen some commits related to npl templates. I thought we had decided to use PROG templates. Even if it is only for testing purposes, I strongly think we should all work with prog templates and only them. This will ease a lot testings for everybody. -- Paul POULAIN et Henri Damien LAURENT Consultants indépendants en logiciels libres et bibliothéconomie (http://www.koha-fr.org)

1 0

utf-8, probable solution
by Paul POULAIN 17 Feb '06

17 Feb '06

Thanks to Heikki Levanto, Tümer Garip & Mike Rylander, you pointed 3 things useless alone, but very useful when mixed. I think I have the solution to our problem. It's not a zebra or html::template or marc::record problem, it's a Perl one ! Let me explain : I followed my utf-8 string in my perl Code until printed and it was always utf-8 (\x9c...) But in firefox, it was iso8859-1. Heikki told me that the first 255 char were shared by unicode and iso8859-1. So, I told myself : OK, Paul, add a "true utf-8 character to your string". I choose \x{263a} (the smiley, because i'm always optimistic & that is what is used in perluniintro) Surprise ... now my é was a utf-8 é in firefox !!!! Conclusion : perl looked at my string before sendint it, and, as it finds it's not "true utf-8", Perl did something to change it in iso8859-1. I also had a brand new message in my log : > Wide character in print at ... Mike R. and Tümer G. suggestions make me investigate perldoc on unicode. and here it is : > A user of Perl does not normally need to know nor care how Perl happens to encode its internal strings, but it becomes rele- > vant when outputting Unicode strings to a stream without a PerlIO layer -- one with the "default" encoding. In such a case, > the raw bytes used internally (the native character set or UTF-8, as appropriate for each string) will be used, and a "Wide > character" warning will be issued if those strings contain a character beyond 0x00FF. > For example, > perl -e 'print "\x{DF}\n", "\x{0100}\x{DF}\n"' > produces a fairly useless mixture of native bytes and UTF-8, as well as a warning: > Wide character in print at ... > To output UTF-8, use the ":utf8" output layer. Prepending > binmode(STDOUT, ":utf8"); > to this sample program ensures that the output is completely UTF-8, and removes the program's warning. GOTCHA ! I have added binmode(STDOUT, ":utf8"), and now, even without the smiley, my éà... are correctly shown. Still having to investigate mySQL utf-8, but it seems that > set NAMES=utf8 is useless. Thanks everybody for helping me. I'll continue this thread on koha-devel only, as zebra & perl4lib are not interested probably. -- Paul POULAIN et Henri Damien LAURENT Consultants indépendants en logiciels libres et bibliothéconomie (http://www.koha-fr.org)

2 2

Re:[Koha-devel] BIG BIG problem (utf8 & mySQL)
by Tümer Garip 17 Feb '06

17 Feb '06

Hi, Our production system koha is utf8 on a windows platform. I do not know whether its different in Unix but here is the solution we found. Any read-writes to mysql is through biblio.pm so we saved the biblio.pm as an utf8 file. (Windows gives you the opportunity of saving a text file in ANSI(iso8859), utf8 or unicode(16bit).) This sorted out 2 problems: 1-Reading,writing from db and display is allways utf8 2- Char_decode subroutine works correctly when importing MARC into Koha I believe what this does is something similar to Pauls smiley solution simply and effectively. With this we did not even need binmode(STDOUT, ":utf8") either which is very cleverly debugged by Paul. Hope it helps, Tumer Garip >Paul wrote: >Hello, > > >(I cc: mike Rylander in case the pines guys have another solution...) > >I continue investigating the utf8 stuff. I have a very very bad feeling : it seems that Perl DBI::mysql does not handle >utf8 correctly. >Here are some links : >http://www.cpanforum.com/threads/654 >http://lists.mysql.com/perl/3714 >http://marc.theaimsgroup.com/?l=msql-mysql-modules&m=111970179409036&w= 2 >http://lists.mysql.com/perl/3006?f=plain > >If I read correctly, there are only 2 ways to solve this : >* bug DBD::mysql maintainer to have the fix included. Would require an official release, an upgrade of DBD::mysql, but >it's a better solution in the long term. >* modify EVERY SQL query to Encode::decode_utf8() every value. A quite huge task ! > >I have modified one script (marc_subfield_structure.pl), and everything works fine with Encode::decode_utf8 (and don't >>work without it) -- >Paul POULAIN et Henri Damien LAURENT >Consultants indépendants >en logiciels libres et bibliothéconomie (http://www.koha-fr.org)

2 1

new features for borrowers
by Paul POULAIN 17 Feb '06

17 Feb '06

Hello all, (& specifically joshua, the RM) I was yesterday in OUEST-PROVENCE (called OP in this mail) to work with them on their version of Koha. We all agree on the following position : developp as many features as possible to be in official CVS. Even if it means developping a little bit more parameters tables, to have a behaviour that anyone can use. They worked on borrowers management, here is what they have done and will do in the next 2 weeks : * in koha 2.x we had 3 kinds of borrowers : Adults, Institutions, Childrens. Those 3 kinds of borrowers where hardcoded (A, I, C), and not fully functionnal. OP has defined a new row in categories table : "cateogy_type". a category_type can be <A>dult, <C>hildren, <I>nstitution, <P>rofessional. The library can thus define in the database which catagories are what (and we can have more than 1 category for <C> for example : Baby, boy, girl,...) * when a library want to add a borrower, it 1st must select what kind of borrower it want to add (A,I,C,P). Then the "member add" begins. Every category_type has it's own template. Add is divided in 3 parts : PART 1 : general information : name, surname (where applicable), category (baby, boy, girl in my previous sample, if you choose category_type='C'), and *city* The city refers to a new table with city/zipcode. The library can fill or ignore this table. If empty, the list does not appear. Otherwise, the list appear. Useful to put usefull a list of common cities your library has. PART 2 : addresses (personnal, professionnal, parent's one...), including mail, fax, ... In the previous part, if a city has been choosen, zip and city name are automatically filled and can be modified if necessary. The address has been modified to be separated in 3 parts : number, kind of road (avenue, road, place...), road name. The list of "road type" is defined in a separate table. If empty, once again, it does not appear. Separating address in 3 parts will be a useful thing to have Geographic information on borrowers. OP plans to connect Koha to a Geographic Information Système (SIG in french) on the long term. PART 3 : library use (opac note...) nothing new or important here, if I don't mind. On every PART value checking is done on client side (new behaviour, in javascript) AND server side (as previously). What is new and could be improved later is that the list of mandatory fields can be defined in a systempreference (zipcode is mandatory in France, for example, it's not is some countries) Last note : OP will also reorganize the borrowers table to normalize fields names. For instance, there are many addresses/zipcode..., each having it's own "nick". I suggested to adopt the following rules : each address/mail/phone/fax information begins with a A or a B, the A being "the most important one", the B being the alternative one. For example : Aaddress1, Aaddress2, Azipcode, Acity... Last note (2) : adding childrens : I suggested to add 3 ways to add a children : - from the parent, add a children = in this case, the guarantorcode is kept, and the B adress is automatically set to the parent one. - create a children from home page, and in PART 2, add a ... button to find the parent to set guarantorcode (and B address) - create a children that has no parent in the database (no guarantorcode, and thus no B address automatically set). This last behaviour being usefull for OP, as childs can subscribe to the library without their parents being members. Expected timeline : code working in around 3 weeks (next week being holidays for OP developpers), but on default templates. We agree to let me install a copy on bureau.paulpoulain.com to let you see how it works, and commit code if everybody agrees. Let me know your opinion on this. Joshua, I let you update release notes for 3.0 -- Paul POULAIN et Henri Damien LAURENT Consultants indépendants en logiciels libres et bibliothéconomie (http://www.koha-fr.org)

1 0

yahoo tools
by Paul POULAIN 15 Feb '06

15 Feb '06

Hello, Yahoo just released (with a BSD license) 2 pieces of code for rich applications : http://developer.yahoo.net/ypatterns/ http://developer.yahoo.net/yui/ I think we should definetly look at them. (the ypatterns contains very useful & dev-friendly things imho : the solution to very common problems) -- Paul POULAIN et Henri Damien LAURENT Consultants indépendants en logiciels libres et bibliothéconomie (http://www.koha-fr.org)

1 0

Koha 3.0 Meeting
by Joshua Ferraro 14 Feb '06

14 Feb '06

Hi all, I've scheduled another Koha 3.0 meeting to be held on Monday Feb 20th at 20:00GMT. Here's the time in your area: http://tinyurl.com/7mztb Our main focus will be the perl-zoom integration, but if you have other topics to discuss, feel free to add them to the agenda: http://www.saas.nsw.edu.au/koha_wiki/index.php?page=AgendAndNotes06feb20 Cheers, -- Joshua Ferraro VENDOR SERVICES FOR OPEN-SOURCE SOFTWARE President, Technology migration, training, maintenance, support LibLime Featuring Koha Open-Source ILS jmf(a)liblime.com |Full Demos at http://liblime.com/koha |1(888)KohaILS

1 0

who knows ?
by Paul POULAIN 14 Feb '06

14 Feb '06

who is http://www.gokoha.com/ ? -- Paul POULAIN et Henri Damien LAURENT Consultants indépendants en logiciels libres et bibliothéconomie (http://www.koha-fr.org)

2 2

Koha 3.0 update for week ending Feb 11th 2006
by Joshua Ferraro 13 Feb '06

13 Feb '06

Hi all, Sorry I'm late getting last week's 3.0 Update out ... better late than never I suppose :-). Last week was especially exciting because we've finally got Perl-ZOOM in a working state -- many many thanks to Paul and the Index Data guys for all their hard work on this. MEETING ANNOUNCEMENT We'll be holding our weekly organizational meeting on Monday, February 13th at 19:00 GMT. Here's the time in your area: http://tinyurl.com/7bk2f MEETING NOTES (for Monday February 6th) Present: kados, owen, thd, sanspach, slef, hdl hdl announced that paul had solved 'problem #2' with perl-zoom, but problem #1 still remained (CQL queries) hdl also announced some new serials code he was working on to link a serial to an item record kados announced some code he committed to rel_2_2, several new sysprefs, some related to Amazon.com content, some to interface customization. group discussed some areas where Koha needed improvement to use mod_perl2 thd brought up his suggestion about improving framework for more SQL dbs: http://lists.nongnu.org/archive/html/koha-devel/2006-02/msg00020.html slef, kados, hdl agreed his plan is sound. still waiting for feedback from paul and chris before adding these suggestions to the Koha Coding Guidelines WHAT'S ON TAP FOR THIS WEEK paul: integrating perl-zoom and biblio functions hdl: Acquisitions chris: get keyword searching going with perl-zoom DEVELOPER ACTIVITY LAST WEEK paul: worked on perl-zoom hdl: Serialsitemization (in rel_2_2) Add a YesNo system preference serialsadditem If set to 1, each issue received will create a new item linked to the biblio with itemnotes filled with Serial number information. chris: testing mod_perl2 with head, ironing out some bugs planning new search module using Perl-ZOOM Cheers, -- Joshua Ferraro VENDOR SERVICES FOR OPEN-SOURCE SOFTWARE President, Technology migration, training, maintenance, support LibLime Featuring Koha Open-Source ILS jmf(a)liblime.com |Full Demos at http://liblime.com/koha |1(888)KohaILS

2 2

testing tool ?
by Paul POULAIN 13 Feb '06

13 Feb '06

Hi, could http://openqa.org/selenium-ide/ be something interesting for us ? -- Paul POULAIN et Henri Damien LAURENT Consultants indépendants en logiciels libres et bibliothéconomie (http://www.koha-fr.org)

1 0