Hi everyone, In case you haven't been following the IRC logs we've been discussing Zebra as a potential searching engine. From Indexdata's website: Zebra is a high-performance, general-purpose structured text indexing and retrieval engine. It reads structured records in a variety of input formats (eg. email, XML, MARC) and allows access to them through exact boolean search expressions and relevance-ranked free-text queries. Zebra supports large databases (more than ten gigabytes of data, tens of millions of records). It supports incremental, safe database updates on live systems. You can access data stored in Zebra using a variety of Index Data tools (eg. YAZ and PHP/YAZ) as well as commercial and freeware Z39.50 clients and toolkits. http://indexdata.dk/zebra I've setup a zebra test site running on LibLime's server. It currently has access to three Zebra datasets, Nelsonville's 150K records, LibLime's 5 million records (recently donated by sanspach), and Paul Poulain's 13K records. (Paul is still working out some issues with indexing unimarc records so stay tuned for that one to work). http://liblime.com/zap/advanced.html Note that the search and retrieval is done via the Z39.50 protocol with the server that ships with Zebra and both the index and the server can be customized based on the kinds of searches you want to perform (the above site is just a proof of concept) -- we'd have support for relevence ranking, stemming, the whole gambit of searching technologies. In all my tests searches are returned in under a second. If we decide to work with Zebra we will need to decide what to do with non-marc libraries. Should we develop an export utility that will allow Zebra to index the records (in say, XML format)? Should we use the Koha tables to create a basic MARC record for use with Zebra? Should we leave the Koha 1.x searching methods unchanged and only use Zebra for MARC libraries? Also, what should we do with the existing marc_*_table tables? So ... it's clearly time to schedule a "Koha 2.4 Searching Group Meeting" on IRC. I'd like to pick a time when everyone can be represented. how is Thursday, June 23 at 9:00 GMT? Here's the time in your area: http://tinyurl.com/925c8 Please let me know on-list if you will not be able to attend and what time you can attend. Comments, suggestions, concerns? -- Joshua Ferraro VENDOR SERVICES FOR OPEN-SOURCE SOFTWARE President, Technology migration, training, maintenance, support LibLime Koha ILS, Mambo Intranet, DiscrimiNet Filter jmf@liblime.com | Full Demos at http://liblime.com | 1(888)KohaILS
Joshua Ferraro a écrit :
If we decide to work with Zebra we will need to decide what to do with non-marc libraries. Should we develop an export utility that will allow Zebra to index the records (in say, XML format)? Should we use the Koha tables to create a basic MARC record for use with Zebra? Should we leave the Koha 1.x searching methods unchanged and only use Zebra for MARC libraries? Also, what should we do with the existing marc_*_table tables? I just add 1 question : zebra is quite simple to install, quite complex to configure, so : do we want a Koha harder to setup or have 2 differents ways to install Koha 2.4[*]
So ... it's clearly time to schedule a "Koha 2.4 Searching Group Meeting" on IRC. I'd like to pick a time when everyone can be represented. how is Thursday, June 23 at 9:00 GMT? Here's the time in your area: http://tinyurl.com/925c8
I think i must be on this meeting. And I want to be there. But june 23 is impossible for me. Next week, 20 and 21 are the only possible days. The week after will be harder (almost impossible). (note that friday 17 is OK for me, as well as thursday 16.)
Please let me know on-list if you will not be able to attend and what time you can attend.
[*] my opinion on Koha 2.4 is ... that is should be called 3.0, seeing all joshua ideas ;-) -- Paul POULAIN Consultant indépendant en logiciels libres responsable francophone de koha (SIGB libre http://www.koha-fr.org)
Joshua Ferraro <jmf@liblime.com> wrote:
In case you haven't been following the IRC logs we've been discussing Zebra as a potential searching engine. [...]
Comments, suggestions, concerns?
Is this for indexing the koha holdings only, or can it be used to talk to the outside world? What do zebra searches look like to the user? I admit, I took a look at the test a while ago when on IRC and didn't get far. Thanks, -- MJ Ray (slef), K. Lynn, England, email see http://mjr.towers.org.uk/ http://www.ttllp.co.uk/koha/
On Tue, Jun 14, 2005 at 11:19:06AM +0100, MJ Ray wrote:
Joshua Ferraro <jmf@liblime.com> wrote: Is this for indexing the koha holdings only, or can it be used to talk to the outside world? Zebra can fully index a MARC record; it also runs as a Z39.50 server at a high level of compliance to the protocol (support for ranking and stemming).
What do zebra searches look like to the user? I admit, I took a look at the test a while ago when on IRC and didn't get far. Zebra searches now look horrible to the user -- but when we package Zebra as the backend biblio retrieval engine it's transparent to the user (looks identical to our current system). But it gets more complex because with Zebra we're already pulling out the whole marc record when we do a search, so we should really not discard everything but the biblionumber. So the question becomes: what is MARC good for and what is Koha good for?
-- Joshua Ferraro VENDOR SERVICES FOR OPEN-SOURCE SOFTWARE President, Technology migration, training, maintenance, support LibLime Koha ILS, Mambo Intranet, DiscrimiNet Filter jmf@liblime.com | Full Demos at http://liblime.com | 1(888)KohaILS
On Tue, Jun 14, 2005 at 11:19:06AM +0100, MJ Ray wrote:
Joshua Ferraro <jmf@liblime.com> wrote: Is this for indexing the koha holdings only, or can it be used to talk to the outside world? Zebra can fully index a MARC record; it also runs as a Z39.50 server at a high level of compliance to the protocol (support for ranking and stemming).
Think of Z39.50 as HTTP "GET" for the library world. That's both inaccurate and over simplfied, but is the gist. One advantage is that it allows users to choose their own clients to search in the library catalog or import directly into their own bibliography manager. (e.g. BookWhere, EndNote, and Procite) Another advantage is that a client can handle multiple Z39.50 targets for cross-searching in multiple catalogs. So if a Z39.50 client is added to Koha AND it is planned right, then it is possible for Koha to offer cross searching in multiple catalogs. Or a remote use could search in their own catalog + Koha.
What do zebra searches look like to the user? I admit, I took a look at the test a while ago when on IRC and didn't get far.
Z39.50 search results are meant to be processed by the client and then displayed for formatting. Since Zebra can return XML, one option is to use XSLT. However, event if you use plain MARC21, then you will still need Koha to do some formatting. I used to do a fair amount of work on customizing web-based front ends that used Zebra and have installed and configured it Zebra quite a few times, but the last time was 3 years ago. If you're used to Z39.50 and MARC, then even the configuration is not so hard. If it's not already been mentioned, there are some overviews of Z39.50 here: http://www.niso.org/z39.50/z3950.html#info By the way, is there an internal API for Koha? -Lars ---------------------------------------------------------------- This mail sent through L-secure: http://www.l-secure.net/
participants (4)
-
Joshua Ferraro -
lars＠rovaniemelainen.com -
MJ Ray -
Paul POULAIN