August 2006 - Koha-devel - lists.koha-community.org

UTF-8 problems : a summary and some solutions
by Henri-Damien LAURENT 23 Aug '06

23 Aug '06

Hi, I investigated quite hard into finding a solution to our utf-8 problems. It seems we CAN come to a solution. What is the problem ? ------------------------------ On my right : values from xml records utf-8 encoded. Correctly displaying them is only a matter of making PERL aware of the fact that he is using utf-8 values, that is : - either use perl -CS or perl -CIOE instead of just simple perl on the first line of the script. That toggles perl into utf-8 for STDINPUT, STDOUTPUT and STDERR. - or add binmode STDOUT, ":UTF-8" or binmore STDOUT, ":utf8" (Be warned that those two different descriptions ARE NOT equivalent. see http://perldoc.perl.org/Encode.html#UTF-8-vs.-utf8 I would suggest that we always use utf8 rather than UTF-8) Simple, clear, efficient (remember though to use libxml2 as stated Joshua in a previous mail http://www.nntp.perl.org/group/perl.perl4lib/2369) On my left : values from Mysql coded in utf-8 and stored as such in MysqlDatabase. When using ("SET names=UTF-8") on the database just after the connection, Everything goes... WELL !!!! Bibliothèque   ¥ · £ · € · $ · ¢ · ₡ · ₢ · ₣ · ₤ · ₥ · ₦ · ₧ · ₨ · ₩ · ₪ · ₫ · ₭ · ₮ · So What is the problem ???? The problem is that left goes well PROVIDED that you DONOT set binmode STDOUT,":utf8" and PERL believes he is processing ISO-8859-1 data. If you launch test_mix.pl on your system without perl -CS : xml data wil be badly displayed BUT Mysql data Well displayed !!! <datafield tag="200" ind1=" " ind2=" "> <subfield code="a">(L') �ole des p�es</subfield> <subfield code="b">R</subfield> <subfield code="f">Bazin, Herv�/subfield> </datafield> <datafield tag="210" ind1=" " ind2=" "> <subfield code="c">�itions du seuil</subfield> </datafield> If you launch test_mix.pl on your system with perl -CS : xml data will be well displayed BUT Mysql data badly (encoded TWICE) !!! BibliothÃ¨que   Â¥ Â· Â£ Â· â¬ Â· $ Â· Â¢ Â· â¡ Â· â¢ Â· â£ Â· â¤ Â· â¥ Â· â¦ Â· â§ Â· â¨ Â· â© Â· âª Â· â« Â· â Â· â® Â· OK. we've got a problem. Where does it come from ? It comes from DBD::mysql and DBI which provides utf-8 data to PERL without Telling him. In perl, Utf-8 data has a flag that tells OK, I am utf-8. If this flag is not set and output is UTF-8 Perl will "magically" but unfortunately re-encode the values. See http://www.mail-archive.com/dbi-dev@perl.org/msg04319.html and following... Solutions. ------------- you can have a look at http://www.simplicidade.org/notes/archives/2005/12/utf8_and_dbdmys.html There are three solutions : - find out EVERYWHERE in the code we receive data from mysql and set the flag on |if (! utf8::is_utf8($message)) { utf8::decode($message); }| Is there a volunteer ? ;) - patch DBD-MYSQL code so that it makes perl utf-8 aware and use {*"*mysql_enable_utf8*"*=>1} options when connecting to database. There is a patch here (version 2.90008 http://lists.mysql.com/perl/3563 and here http://rt.cpan.org/Public/Bug/Display.html?id=17829 version 3.000006 ) And it seems that people in the DBD-MYSQL team will incorporate this patch in next DBD::mysql version. - Use (http://dysphoria.net/2006/02/05/utf-8-a-go-go/) UTF8DBI.pm for any database connections rather than DBD-mysql. Advantage : If we incorporate those package tightly to C4 and Koha, we may improve UTF-8 compliance and stability. Disadvantages : Module is not supported by anyone, not official, and would require quite HEAVY transformations in our code. You may have felt which solution seems to me the most reasonale one. The only problem being... Are we finished with utf-8 problems ? I HOPE So. But we can reasonably think there still may be hidden ones. Why ? - http://www.mhonarc.org/archive/html/perl-unicode/2006-07/msg00004.html and http://www.mhonarc.org/archive/html/perl-unicode/2006-07/msg00008.html seems to point out some CGI input problems with UTF-8. (It is a fact I DIDNOT test data input with binmode UTF-8 set or with the DBD-MYSQL hack.) So I go on testing. If you need test files, you can get them on http://hdlaurent.paulpoulain.com/test_mix.pl http://hdlaurent.paulpoulain.com/test_mix_fixed.pl http://hdlaurent.paulpoulain.com/testsrecord.xml -- Henri-Damien LAURENT

2 4

Adding item e - niveau de localisation
by Pedro Faria 23 Aug '06

23 Aug '06

Dear all, I'm trying to add an item to Koha, from a MARC record via z3950 searches. It works fine till the moment of filling in mandatory fields in .../cgi-bin/koha/acqui.simple/additem.pl?bibid=6&frameworkcode= there is one, namely e - niveau de localisation which does not have any value to choose from? What is this value? Where can I add values to that dropdown box? Other question? Why are some templates in french, even when I choose only en language?

1 0

Testing Z3950 searches
by Pedro Faria 22 Aug '06

22 Aug '06

Dear all, I am having serious problems with Z3950 searches. I'm trying to make some searches on pre-added z3050 searches but it keeps giving No results found when it should give some results! I read something about a deamon which is in koha/intranet/scripts/z3950daemon/ . Can someone explain me what I need to do to set this up?

1 0

Re: Re Re: [Koha] Koha reports not working
by Mohamed Imran K R 22 Aug '06

22 Aug '06

Hi i have attached the Auth.pm file. pls look into the same and solve my ldap query the problem it does let the users in to opac, but it gives out a warning in the logs as [Tue Aug 22 14:54:29 2006] [error] [client 192.168.1.2] LDAP Auth failed server not responding or wrong user password [Tue Aug 22 14:54:29 2006] [error] [client 192.168.1.2] combination at /usr/local/koha/intranet/modules/C4/Auth.pm line 529. i checked the members section on the lib interface and it has indeed added a user. also i am not able to fetch any data from my ldap server and populate the koha db pls do look into this On 8/22/06, Chris Cormack < chris(a)katipo.co.nz> wrote: > > Henri-Damien LAURENT wrote: > > Mohamed Imran K R a écrit : > >> there are thousands of users and more provisioned. so i just want the > >> changes to be reflected as and when the users change info about > >> themselves. for manual sol, i have to synchronize every now & then > >> hoping that there is no loss. > >> > >> the koha ldap module fetches & fills in the data from the ldap into > >> the borrower table of koha db. i dont how is it related to MARC! > > It is not. > > koha_ldap modules, as far as I know, was designed by someone who needed > > it once, neither by me nor by paul (iirc). And it seems NOT populating > > borrowers table with data from LDAP. LDAP is merely used as a login > > facility. > > > > > > But paul will come back next week I can ask him. > Hmmm > > Looking around line 575 it seems that it updates the borrowers table, or > creates a new borrower if one doesnt exist. > > But we should probably continue this discussion on the koha-devel list. > > Chris > -- Regards Mohamed Imran K R

2 2

ALL Programmers & Template designers please note
by Tumer Garip 22 Aug '06

22 Aug '06

Hi all, Although officially KOHA runs on Unix and Firefox there is quite a lot out there that uses Windows and Internet Explorer with Koha. Especially with the new ZOOM integration KOHA is now fully Windows compatible as well. However a silly javascript error prevents current frameworks from being used in Internet Explorer. The bug (I consider it a bug in IE) is that any popup window name that contains a space throws a javascript error. So please name the popup windows in window.open scripts with no space. i.e. instead of “Koha Help” pls use “Koha_Help” or a single word. This will help those windows users like us as not to edit all the templates everytime we update. I know that officially we say Firefox but I think it’s a shame to ignore IE just because of one its bugs. Thanks all, Tumer -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.1.405 / Virus Database: 268.11.3/423 - Release Date: 18.08.2006

3 2

New feature (idea) for Koha 3
by Paul POULAIN 16 Aug '06

16 Aug '06

Hello, A new feature that could (should ?) be implemented in Koha 3 : biblios have the "ISBD" system preference, that contains a personnalised version of the biblio. My libraries reported that it would be nice (better) to have : 1- 1 specific "template" for each itemtype 2- a specific "template" to show the summary of the biblio in the result list (= having a different template for monographies & serials for example) 3- the same thing for authorities. (I must admit that authorities details are really awful, it's almost the raw MARC record) Unless someone declares he is already working on this, I'll do it (me or hdl or antoine, depends on who get some time to do this. I think 2 days are needed to do something working well) PS : Other ideas in the same area : - rewrite the "normal" (=detail.pl), to get rid of non MARC database - add a systempreference to enable MARC view or disable it in OPAC. Warning : "template" don't mean "HTML::Template", it means something the library can modify -- Paul POULAIN et Henri Damien LAURENT Consultants indépendants en logiciels libres et bibliothéconomie (http://www.koha-fr.org) Tel : 04 91 31 45 19

4 4

FW: RDBMS like behaviour or merged result sets from ZEBRA-ZOOM
by Tümer Garip 16 Aug '06

16 Aug '06

-----Original Message----- From: Mike Taylor [mailto:mike@miketaylor.org.uk] Sent: Tuesday, July 25, 2006 12:43 PM To: Tümer Garip Cc: support(a)indexdata.dk; 'Joshua Ferraro' Subject: RDBMS like behaviour or merged result sets from ZEBRA-ZOOM Tümer Garip writes: > We have come a long way in integrating ZEBRA and ZOOM into KOHA and > first examples are already running in some libraries. That's great news. > However we now have a problem that we could not solve and wonder > whether its us or the capabilities of ZEBRA or ZOOM (probably all) > thats preventing further development. > > We have two different types of MARC records. Namely bibliographic > and holdings. These records are related to each other with uniquie > ID numbers. A bibliographic record can have multiple holdings > record related to it. > > We are having problem searching these records in RDBMS manner. i.e > Searching for say title which exists in bibliographic record and > limiting that search to a specific branch which exists in holdings > record. Let me make sure I understand what you're trying to do here. It's a two-stage search, in which you first want to search the list of works for books with a certain title, then find the holdings records for nearby libraries that have the books of that title? In a relational database, it would be something like this: select all from works,holdings where works.title = 'Harry Potter' and holdings.work_id = works.id and holdings.library_id in (45,76,102); Correct? If so, you simply cannot do this with Zebra -- it's a relational query and Zebra is not a relational database. You can't even express such a query in the Z39.50 Type-1 or SRU CQL query languages. > We tried putting all the records in the same database could not do > it. We than tried putting them in different databases (even > different servers) but could not find a way of telling zebra to > merge the results together. > > For a small resultset we can do a recursive search of ID's on the > other database and get a merged resultset but library operations > work on large numbers and that is impossible when the small set is > around 1800 records. I think you mean code like this? (WARNING: UNTESTED) $rs = $works_conn->search('title="harry potter"); foreach my $i (1 .. $rs->size()) { $rec = $rs->record($i); $id = extract_identifier_from_record($rec); $rs2 = $holdings_conn->search("id=$id and library_id=45 or library_id=76 or library_id=102"); push @holdings_records, records_extracted_from($rs2); } That approach is the obvious one to take; but, no, it doesn't scale well when the initial result set is large. > Is there any way we can actually do this with the capabilities of > ZEBRA&ZOOM? > Something like @attr 1=52 (@resultset1 @attr 1=52)? > Or may be a new z:index called linkid which joins two databases or > different records within a database. What you're talking about is certainly in the realm of the possible, but it would be a _significant_ research, design and implementation task -- one, I think, that would beyond the Koha consortium's ability to fund. I am sorry to disappoint, but I think you're going to have to work round this one. > Any idea is appreciated as the whole database design is currently > stuck with this problem. Have you considered adding redundant data, such as author and title, to the holdings records? Then you could search them directly. Not pretty, but it may be pragmatic. _/|_ ___________________________________________________________________ /o ) \/ Mike Taylor <mike(a)miketaylor.org.uk> http://www.miketaylor.org.uk )_v__/\ The Unix command to kill processes is not "kill -9". The Unix command to make a tape archive is not "tar cv". Options are optional.

4 3

Theme Structure 3.0
by MJ Ray 15 Aug '06

15 Aug '06

Our current template structure is something like: SITE/htdocs/SITE-tmpl/THEME/LANG/OBJECT where SITE = { intranet, opac } THEME = { default, npl, vanilla, ... } LANG = { <empty>, en, es, fr, pl, zh-TW, ... } OBJECT = { <empty>, includes, LANG=en?images } It's probably not ideal to have all of these in htdocs, as some aren't really for direct download. Languages and images are also subordinate to themes, which has some side-effects. I suggest splitting includes, languages and templates so we get something like: templates/TYPE/THING/SCOPE where TYPE = { htdocs, l10n, tmpl } THING = { images, includes, LANG, default, npl, vanilla, ... } SCOPE = { default, intranet, opac, ... (may grow if needed) } and language, style and theme are set independently in koha. koha then looks at templates/l10n/LANG:templates/tmpl/THEME :templates/tmpl/THEME/LANG for its files, which may include things from templates/htdocs served by the httpd. This will let koha themes, styles and translations be maintained and used independently. As this requires some rewriting of the httpd config, it should be possible to support the single-vhost installation which seems a FAQ (when you only have one hostname-port combination to run on). The downside is that we will need to provide guidelines for the themes, styles and translations and it may need some careful work to get the same performance as the current template compiler. So that we don't lose anything if it doesn't work, I will branch 3.0 when the time comes and develop/test it there. Comments? Thanks, -- MJ Ray - see http://mjr.towers.org.uk/email.html North End, Lynn, Norfolk, England Work: http://www.ttllp.co.uk/ IRC/Jabber/SIP: on request

6 7

Koha 3, new features local "includes" directory
by Paul POULAIN 14 Aug '06

14 Aug '06

Hello, During the dev_week, in May, we spoke of a new method to have personalized templates. the idea being to have something more easy to changes than a full theme of templates. It's very hard to keep a full set of templates uptodate, the difficulties we have with default & npl on rel_2_2 branch show clearly this ! HTML::Template is really smart : it contains a TMPL_INCLUDE pragma that can be used to import outside templates. The directories where the templates are taken from is defined when you open the template. We have divided our PROG templates in different conceptual parts : * the header, in doc_head_open.inc & doc_head_close.inc (we have 2 so we can put a <title> in it) * the main menu, in menus.inc * the module menu, in menu-modulename.inc. Each module (acquisition, cataloguing, authorities, circulation, members, serials, parameters, tools, reports...) has it's own included file. * the main part of the page : it's specific to each template. * the footer, in intranet-bottom.inc I just have added a systempreference, called "intranet_includes" that let you define the directory where the included files will be taken from. by default, it's the "includes" directory. BUT, if you modify the value to "includes_npl", you'll arrive to something that look like the npl templates (roughly, it's just a Proof of Concept). -- Paul POULAIN et Henri Damien LAURENT Consultants indépendants en logiciels libres et bibliothéconomie (http://www.koha-fr.org) Tel : 04 91 31 45 19

3 2

Upload MARC records in reservoir
by miguel miguelito 13 Aug '06

13 Aug '06

It is the first time that I write to the list, thanks for the great work and aid to all. The concerned registries are stored in the table marc_breeding Now according to description as it looks for a new title or ISBN first looks for in reservoir and if it finds it there passes to the cutting bilio, but to my it does not work to me, talvez so that I am in version Windows and/or lack data, notices that local data do not rise, in the file iso, this file iso creates it according to the manual of koha.org, denominated Migrating to CDS/ISIS based system to koha, concretely of even the Now section the tab separated cases out using to Perl script (shown below) and convert it ace MARC! or UNIMARC. Thanks for rpta. Greetings. --------------------------------- LLama Gratis a cualquier PC del Mundo. Llamadas a fijos y móviles desde 1 céntimo por minuto. http://es.voice.yahoo.com

1 0