- Koha-devel - lists.koha-community.org

Large library
by Ernesto Silva 11 Sep '06

11 Sep '06

Hi, in a relative near future I may be involved in migrating a big library, aprox. 250.000 biblios. I know zebra support is new, but do you think it is possible for zebra to support such a volume of biblios? What about the old mysql and table horizontal partitioning + clustering? Thanks, -- Ing. Ernesto Silva. Coordinador de Desarrollo Web y Sistemas Abiertos Universidad ORT Uruguay. E-mail: silva(a)ort.edu.uy Tel: (+598-2) 902-1505 ext. 206

2 2

Re: [Koha-devel] Large library
by Joshua Ferraro 10 Sep '06

10 Sep '06

On Sun, Sep 10, 2006 at 08:46:26PM -0000, Thomas Dukleth wrote: > On Sun, September 10, 2006 12:54 pm, Joshua Ferraro wrote: > > In fact, Zebra can handle about 50 million records > [snip] > I am expecting Zebra can handle many hundreds of millions of records or my > plan needs rethinking. You are correct Thomas, I responded hastily. I should have said "Zebra can easily handle more than 50 million records". The point being that it is currently in production environments without any specialized hardware with up to 50 million records. Just how scalable it is beyond that point is outside my scope of knowledge; then again, there are really only one or two collections in the known universe with more than 50 million records ;-). Cheers, -- Joshua Ferraro SUPPORT FOR OPEN-SOURCE SOFTWARE President, Technology migration, training, maintenance, support LibLime Featuring Koha Open-Source ILS jmf(a)liblime.com |Full Demos at http://liblime.com/koha |1(888)KohaILS

1 0

Roadmap meeting ?
by Henri-Damien LAURENT 10 Sep '06

10 Sep '06

Hi, it seems here that we have four Koha versions : - 2.2.6 quite stable and published VERY soon. But also : - devweek based version 2.4 which changes search API and is based on zebra. It works with US-MARC and is using some zebra features and it is very appealing as far as facets and OPAC is concerned. We need some explanations in order to make out what Catsearch3 and Catsearch4 are there for. Shall they replace catalogsearch ? Shall they be normalized on catalogsearch API ? What new features they provide ? - rel_3_0 synching branch 2_2 and devweek and all the previous development made. - HEAD which will be based on zebra2.0 and which will highly use XML and XSLT. Those three branches are under development and thus are not stable. The point I am heading to is that Paul and I think it could be great to have an irc meeting quite soon to gather all those parts together and to get a clear information for the status for each branch, therefore make out a general idea of a roadmap/timeline and how those parts will work with one another. And we also need to decide whether to include UTF8DBI.pm and where, which we favor, as Tumer did at first in HEAD and what to do for CGI UTF8 support. To strengthen the project, we should soon join branches so that developments could be shared by the core team and not be wasted for want of time or synching hazards. So, can we schedule an IRC meeting ? -- Henri-Damien LAURENT

3 2

UTF-8 issues, Solution ?
by Henri-Damien LAURENT 05 Sep '06

05 Sep '06

Hi I tried to install HTML::Template::Pro. It seems to fix all our encoding issues. I need to make deeper tests. I commited the changes in Output.pm BUT if it raises new problems for you, please, fell free to revert or comment on that solution. -- Henri-Damien LAURENT

2 1

Koha 2.3.0 problem instaling
by Pedro Faria 05 Sep '06

05 Sep '06

Hi all, I am trying to install koha 2.3.0 but I got the following error: Unable to recognise encoding of this document at /usr/lib/perl5/site_perl/5.8.8/XML/SAX/PurePerl/EncodingDetect.pm line 96. Document requires an element [Ln: 2, Col: 0] BEGIN failed--compilation aborted at scripts/updater/updatedatabase line 21. Problem updating database... Any ideas? XML::SAX is up to date, so says CPAN...Thank you.

2 1

Koha and RSS Feed
by Partha Mukhopadhyay 05 Sep '06

05 Sep '06

Dear Koha developers and users, I would like to know the possibility of RSS feed for Koha. I noticed one rss folder in \intranet\... in both versions of koha (Linux and Windows). I utlized the rss.pl file (after modifying lastAcquired.conf file) to generate recently acquired documents in XML format (the file generated in the name of lastAcquired.xml). This is an excellent feature to auto generate list of recent arrivals for library in RSS feed format. My question is a simple one. Suppose I'm using an open source blog software (for my case it is WordPress), what I should enter as RSS URL for koha in the blog software. Am I clear enough? Pl. advice. Parthasarathi Mukhopadhyay Lecturer, DLISc, Vidyasagar University, India -- __________________________________________________ Now you can search for products and services http://search.mail.com

1 0

QA on Stable Branch
by Joshua Ferraro 31 Aug '06

31 Aug '06

Hi all, Now that pierrick has retired as QA manager ... This email is to remind everyone that the 'stable' branch of Koha is so-named because it's supposed be the most 'bug-free' version of Koha. Especially since we're nearing the release of 2.2.6, it's very important that all commits don't break existing functionality. Please, I can't stress this enough, before you commit something, please test the function completely to make sure it works. Also, test any related functions that may be affected by your commit. This may take a few extra minutes, but it will save the rest of us hours of troubleshooting a so-called 'stable' branch. If you do happen to commit something that breaks Koha, please either fix the bugs immediately, or revert the code back to a working state. There's nothing more annoying than trying to troubleshoot someone else's buggy code. Remember, if you wrote something, it will take you much less time to fix a bug than someone else. As a further reminder, there are still 9 'blocker' bugs that we know about, preventing us from releasing 2.2.6. If you own these bugs, please try to either fix them or revert to the last working copy as soon as possible. (here are the blockers: http://tinyurl.com/fvfxw) Thanks, -- Joshua Ferraro SUPPORT FOR OPEN-SOURCE SOFTWARE President, Technology migration, training, maintenance, support LibLime Featuring Koha Open-Source ILS jmf(a)liblime.com |Full Demos at http://liblime.com/koha |1(888)KohaILS

3 3

rel_3_0 has been created
by Antoine Farnault 30 Aug '06

30 Aug '06

Hi koha-devel, Last week, i have created the rel_3_0 CVS branch and have totaly forgotten to announce it on this list. Shame on me ! (Just spoken about this on IRC) So, this branch - is synchronized with dev_week, and i'll take care to keep it. - containts all the new features provide by zebra, by the SAN, and paul.... etc. - allows tumer to investigate his new features on head. - should be stabilised to be on production for the SAN soon. Cheers, -- Antoine Farnault

1 0

Useful tools for programmers...
by Antoine Farnault 29 Aug '06

29 Aug '06

Hi koha-devel, I've just created a page on the wiki (http://wiki.koha.org/doku.php?id=en:development:tools) to list some usefuls tools for programmers. Don't hesitate to list yours. It can change your life ! ;-) -- Antoine Farnault

1 0

New KOHA API at HEAD
by Tümer Garip 26 Aug '06

26 Aug '06

I have started committing a new API to head and here is what is implemented and what is foreseen. The new DB: changes since 2.2.6; 1-All the information related to biblios, items, and authorities are now held in separate MARC records (currently both MARC and MARCXML and will may drop MARC in due time as well.We have to discuss this!). 2- Biblioitems table completely depreceated. 3- Items table has itemnumber-biblionumber-barcode-marc-marxml-timestamp only 4-Biblio table has biblionumber-title-author-itemtype-isbn-issn-frameworkcode-marc-marcxml- timestamp only 5-auth_header table has authid-authtypecode-marc-datecreated 6- marc_subfield_structure renamed biblios_subfield_structure and kohafield dropped,hidden is tiny int(3) 7- marc_tag_structure renamed biblios_tag_structure 8- biblio_framework renamed biblios_framework 9-New tables added: holdings_tag_structure,holdings_subfield_structure,koha_attr,zebraqueue Concepts behind; 1- Items (holdings) is now a complete record in itself. This provides more flexibility with storing data that was supposed to fit in one field in the early version. Now a whole record. But all the sql fields are gone. So anything you want to put it you have to store them in MARC. Thats why those fields with with %9% in them are designed for. 2- Faster circulation. Only update items record. 3- We can hide the items record from a union catalog in Z3950. Private data in items is protected. 4- Frameworks handling is only for record input and display purposes. You can define different frameworks for biblios and authorities only. Holdings frameworks you can design but cannot rename. A framework name is given at biblios level which creates matching frameworks for biblios and holdings. That is if you create a framework called 'Music Records' then you set it up one for biblios and one for holdings. (Its simpler than it sounds) 5- Visibility flag in frameworks is more verbose. You select "Show","Show collapsed" or "Hide" separately for "OPAC" "Intranet" and "Editor" 6- A new koha_attr table provides the field matching between DB fields and MARC record. Apart from a few fields like biblionumber, itembumber etc. that KOHA requires (and have to be defined to work) the rest is upto the system designer what to match. 7- The same table matches Zebra indexing terms to koha terms or field names. I say kohaterms because most of the fields now do not exist. They are pseudo field names so that we can match 'dewey' to some marc field. In the example koha_attr table provided (for MARC21) all bibliographic fields are matched to Bib-1 attributes to make it universally searchable, while all holdings data is private matches 8- Same table allows creating search screens with field labels and not bothering about what the PQF attibute is for it. All is internally taken care of. You have a show in intranet and show in opac flag. A sort flag is added so that any field that sorts automatically gets added to search screens. 9- zebraqueue table is for zebraupdates. Zebra updating cannot keep up with large libraries circulation demands. So this puts updating in a queue and works asynchronosly so that KOHA can go on doing other things. It has a system preference time switch how long to wait between updates from (0 to forever in seconds). Or a batchMode preference to use shadow files of zebra and not to commit until told so. I personally found zebraqueu to be more reliable than batchMode. New Zebra indexing and KOHA meta record: 1- Having separated holdings record from bibliographic earlier zebraindexing was of no use. Zebra is not a relational DB. (I am surprised at people commenting on this subject at koha-devel !!!). 2- A new KOHA meta record is designed based on MARCXML. Currently it holds one bibliographic and many holdings records in marxml but can be extended to hold authorities records as well. 3- Each individual record is fullfledged MARCXML. 4- The design is <kohacollection><koharecord/></kohacollection>. A koha collection can hold many koha meta records. Its usefull for batch indexing of ZEBRA. Yes we can now batch index XML records with ZEBRA! 5- A koha meta record is <koharecord><record/><holdings><record/></holdings></koharecord> Each <record/> is a MARCXML record. The first one is the bibliographic and any record withing the holdings tag is considered a MARCXML item record.Holdings tag can hold 0 to many records. 6- Zebra can display either the whole koharecord, only the bibliographic or the holdings record. Using yaz-proxy an outside connection to Z3950 server (a union catalogue may be) can be limited to see only the bibliographic record. API and whats next; 1- Due to the urgency of our needed to have the system working ASAP so the current committed API is a mixture of 2_2_6 and 3.0 and working ie it uses both MARC iso2709 and MARCXML. The display and search should be written in xsl stysheets for performace. Already tested it with LibXML perl module and results are promising. 2- Using ZOOM the dependency on Event and Net::Z3950 is depreceated. No more deamons for Z3950 searches. This way KOHA may claim full Windows and IE compatibility. The concept is changed so that cataloger may choose servers on the fly and any record he/she retrieves is not a new addition of biblio but just replaces the old one if he/she was modifying one. 3- HTML::Template is replaced with HTML::Template::Pro - for UTF8 issues 4- A top layer UTF8DBI by Pavel Kudinov http://search.cpan.org/~kudinov/ is implemented as part of API - for UTF8 issues 5- KOHA is now fully UTF8 compliant. 6- Repeated fields and adding authorities bug in addbiblio.pl fixed 7- Using hardcoded subfield 9 for linking authorities to biblios removed. A pseudofield called auth_linkid is defined in koha_attr table which can map to anything with the notation '0009' to use subfield 9 for any tag. Thats generally what it is. I will try to provide an upgrade script from 2_2_6 to HEAD soon. In the meantime once all the commit is finished next week you should be able to play with it with new data. Cheers Tumer

1 0