[Koha-devel] Problems when using Indexdata's Zebra deb packages on Debian Jessie

[David Cook]

Hi all,

As Zeno Tajoli has mentioned on the "Zebra: the 'elements zebra::snippet'"
thread, there are problems using Indexdata's apt repository for Zebra on
Debian Jessie.

It looks to me that the problems arise from YAZ 5.8.1 onwards, although YAZ
5.15.2 is the version at which we're looking.

But the problem might not be with YAZ/Zebra per se. It seems to me that the
problem is actually with our Zebra configuration files. Namely
retrieval-info-bib-dom.xml, but also marc21-retrieval-info-bib-dom.xml,
normarc-retrieval-info-bib-dom.xml, and unimarc-retrieval-info-bib-dom.xml
(in debian/templates), which are used by the Debian packages.

We define many front-end elements using <retrieval syntax="xml"
name="something">, but we don't really need to do that. We can just use
<retrieval syntax="xml" />. That gives us access to the elements we define
in retrieve pipeline in the etc/zebradb/biblios/etc/dom-conf.xml file, as
well as the special retrieval zebra::* elements. 

It seems that <retrieval syntax="xml" name="zebra::*" /> doesn't work with
Zebra 2.0.61 (or I suspect more accurately recent versions of YAZ). I
suspect that Zebra/YAZ is looking in etc/zebradb/biblios/etc/dom-conf.xml
for the element names and failing without realizing that it's actually a
special retrieval element built-in to Zebra/YAZ.

All is fixed if we remove the <retrieval syntax="xml" name="zebra::*" /> and
restart Zebra. Personally, I think we should remove all the <retrieval
syntax="xml" name="something"> lines and just use <retrieval syntax="xml"
/>. That's worked on Zebra 2.0.61/YAZ 5.15.2 on Debian 8, and Zebra
2.0.60/YAZ 5.1.2 on openSUSE 13.2.

I suppose the thing to do now is just open a Bugzilla bug and fix it, but I
thought I'd give a more succinct FYI than I gave in my reply to Zeno
(outlined below).

David Cook
Systems Librarian
Prosentient Systems
72/330 Wattle St, Ultimo, NSW 2007


> -----Original Message-----
> From: David Cook [mailto:dcook at prosentient.com.au]
> Sent: Thursday, 28 January 2016 12:59 PM
> To: 'Tajoli Zeno' <z.tajoli at cineca.it>
> Cc: 'koha-devel' <koha-devel at lists.koha-community.org>; 'Wolfram
> Schneider' <wosch at freebsd.org>
> Subject: RE: [Koha-devel] [SOLVED] Zebra: the 'elements zebra::snippet'
> doesn't work
> 
> Hi Zeno:
> 
> I believe that I've solved the symptom, although I admit that I don't 100%
> understand the root casue of the problem.
> 
> The problem appears to be the line '<retrieval syntax="xml"
> name="zebra::*" />' in "etc/zebradb/retrieval-info-bib-dom.xml".
> 
> If you remove that and restart Zebra for your instance, you'll find that
you're
> once again able to use "zebra::index", "zebra::snippet", "zebra::data",
> "zebra::facet::index:register:count".
> 
> The in-depth explanation starts here:
> 
> The vital line really is '<retrieval syntax="xml"/>'. With that line, we
have
> access to the special retrieval elements
> (http://www.indexdata.com/zebra/doc/special-retrieval.html), as well as
> the ones that we've defined in the "retrieve pipeline" in
> /etc/koha/zebradb/biblios/etc/dom-config.xml, such as "zebra", "index",
> "marc", and "marcxml". (See /etc/koha/sites/libraryname/zebra-biblios-
> dom.cfg for the reference to /etc/koha/zebradb/biblios/etc/dom-
> config.xml. Also see http://www.indexdata.com/zebra/doc/record-model-
> domxml-pipeline.html for an explanation of the pipeline: "The possible
> multiple <retrieve> pipeline definitions are distinguished by their unique
> name attributes, these are the literal schema or element set names used in
> SRW, SRU and Z39.50 protocol queries.").
> 
> I'm going to guess that the problem arose with YAZ 5.8.1 2015/01/13:
> "retrieval: pick matched element-set rule YAZ-813"
> (http://www.indexdata.com/yaz/doc/NEWS). Here's the git commit:
> http://git.indexdata.com/?p=yaz-moved-to-
> github.git;a=commit;h=f24766f1e9fc5404fc0b512af8607d7f7054f4be. I
> haven't read through it extensively, so I could be wrong, but perhaps it
> caused some sort of problem where Zebra/YAZ was trying to find "zebra::*"
> in /etc/koha/zebradb/biblios/etc/dom-config.xml. I'm guessing that it
didn't
> check that it was a special retrieval element and that it failed to find
the
> backend element in the retrieve pipeline, and that's why it failed.
> 
> We don't actually need any of the following lines:
> <retrieval syntax="xml" name="index" />
>   <retrieval syntax="xml" name="marc"
>              identifier="info:srw/schema/1/marcxml-v1.1"/>
>   <retrieval syntax="xml" name="marcxml"
>              identifier="info:srw/schema/1/marcxml-v1.1"/>
>  <retrieval syntax="xml" name="zebra::*"  />
> 
> The only thing we need is <retrieval syntax="xml"/>. That provides access
to
> the "marc", "marcxml", "zebra", and "index" backends defined in
> /etc/koha/zebradb/biblios/etc/dom-config.xml, as well as the zebra::*
> special retrieval elements.
> 
> I've actually tested that on YAZ 5.1.2 and Zebra 2.0.60 on openSUSE, and
it
> works great.
> 
> I suppose the thing to do at this point is open a Bugzilla report to
change
> "etc/zebradb/retrieval-info-bib-dom.xml", as well as
"marc21-retrieval-info-
> bib-dom.xml", "normarc-retrieval-info-bib-dom.xml", and "unimarc-
> retrieval-info-bib-dom.xml".
> 
> --
> Installed Koha and Zebra on Debian Jessie using packages from scratch:
> sudo ls
> wget -O- http://debian.koha-community.org/koha/gpg.asc | sudo apt-key
> add - echo 'deb http://debian.koha-community.org/koha oldstable main' |
> sudo tee /etc/apt/sources.list.d/koha.list sudo apt-get update sudo
apt-get
> install koha-common
> --
> sudo apt-get install mysql-server
> <set mysql root password>
> <edit /etc/koha/koha-sites.conf : INTRAPORT="8080"> sudo a2enmod
> rewrite sudo a2enmod cgi sudo service apache2 restart echo 'deb
> http://ftp.indexdata.dk/debian jessie main' | sudo tee
> /etc/apt/sources.list.d/indexdata.list
> sudo apt-get update
> --
> sudo apt-get install libidzebra-2.0
> 
> sudo koha-create --create-db libraryname sudo a2dissite 000-default.conf
> sudo vi /etc/apache2/ports.conf <add Listen 8080> sudo service apache2
> restart
> 
> sudo xmlstarlet sel -t -v 'yazgfs/config/user'
> /etc/koha/sites/libraryname/koha-conf.xml
> sudo xmlstarlet sel -t -v 'yazgfs/config/pass'
> /etc/koha/sites/libraryname/koha-conf.xml
> 
> <choose MARC21 and all sample data>
> <create administrator patron>
> <create MARC record>
> <worth noting that it appears to use CHR indexing by default>
> 
> yaz-client unix:/var/run/koha/libraryname/bibliosocket
> base biblios
> find test
> [109] Database unavailable -- v2 addinfo 'biblios'
> <wait a few seconds>
> base biblios
> find test
> Number of hits: 1, setno 3
> show 1
> <all good>
> format xml
> show 1
> <all good>
> elements zebra::snippet
> show 1
> [25] Specified element set name not valid for specified database -- v2
> addinfo ''
> elements index
> show 1
> <all good>
> elements marcxml
> ehow 1
> <all good>
> elements marc
> <all good>
> --
> elements zebra
> show 1
> [biblios]Record type: XML
> <?xml version="1.0" encoding="UTF-8"?>
> <z:info xmlns:z="http://indexdata.com/zebra-2.0" z:id="2"
> z:filename="/tmp/yIjODFCwAP/upd_biblio/exported_records" z:rank="0"
> z:score="" z:schema="zebra" z:size="938"/>
> --
> sudo vi /etc/koha/marc21-retrieval-info-bib-dom.xml
> <remove "zebra::*" entry>
> sudo koha-restart-zebra libraryname
> 
> Z> elements zebra::snippet
> Z> show 1
> Sent presentRequest (1+1).
> Records: 1
> Record type: XML
> <record xmlns="http://www.indexdata.com/zebra/">
>   <snippet name="Any" type="w"><s>test</s></snippet>
>   <snippet name="Any" type="w">This is a <s>test</s></snippet>
> </record>nextResultSetPosition = 2
> Elapsed: 0.001125
> 
> Z> elements zebra::facet::title:w
> Z> show 1
> Sent presentRequest (1+1).
> Records: 1
> Record type: XML
> <record xmlns="http://www.indexdata.com/zebra/">
>   <facet type="w" index="title">
>     <term coccur="2" occur="1">a</term>
>     <term coccur="2" occur="1">is</term>
>     <term coccur="2" occur="1">test</term>
>     <term coccur="2" occur="1">this</term>
>   </facet>
> </record>
> 
> I had suspected in the past that we might be re-defining retrieval
elements in
> the /etc/koha/marc21-retrieval-info-bib-dom.xml file... and this appears
to
> confirm it.
> 
> 
> 
> David Cook
> Systems Librarian
> Prosentient Systems
> 72/330 Wattle St, Ultimo, NSW 2007
> 
> 
> > -----Original Message-----
> > From: Tajoli Zeno [mailto:z.tajoli at cineca.it]
> > Sent: Thursday, 28 January 2016 4:16 AM
> > To: David Cook <dcook at prosentient.com.au>
> > Cc: koha-devel <koha-devel at lists.koha-community.org>
> > Subject: Re: [Koha-devel] Zebra: the 'elements zebra::snippet' doesn't
> > work
> >
> > Hi David and all,
> >
> > Il 27/01/2016 06:19, Zeno Tajoli ha scritto:
> > >> Did you load an old Koha configuration and database? Or did you
> > >> start
> > fresh?
> > >> You might need to do a Zebra re-index and a Zebra re-start.
> > >
> > > It is a start from fresh install and I do same  Zebra re-index and a
> > > Zebra re-
> > start.
> > > But now I try starting from a fresh debian 8 and I will say the
result.
> >
> > I finish my test about 'elements zebra::snippet'.
> > I have done 4 different installation on debian 8 (jessie) In every case:
> > Install type: koha-common
> > OS: Debian 8 (jessie)
> > Lang: en
> > Koha versione: 3.20.7.1 (oldstable)
> >
> > 1)Zebra: 2.0.61. Repos: Indexdata and Koha. MARC type: MARC21
> > -- 'elements zebra::snippet' doesn't work
> >
> > 2)Zebra: 2.0.61. Repos: Indexdata and Koha. MARC type: UNIMARC
> > -- 'elements zebra::snippet' doesn't work
> >
> > 3)Zebra: 2.0.59. Repos: Koha only. MARC type: MARC21
> > -- 'elements zebra::snippet' WORKS
> >
> > 4)Zebra: 2.0.59. Repos: Koha only. MARC type: UNIMARC
> > -- 'elements zebra::snippet' WORKS
> >
> > So the problem is not connect with MARC21/UNIMARC.
> >
> > David says that on OpenSuse Zebra 2.0.60 is ok about 'elements
> > zebra::snippet'.
> >
> > It is not clear if it is a Debian 8 problem or an error on Zebra
> > 2.0.61
> >
> > I try to ask in Zebra mailing list.
> >
> > Bye
> > Zeno Tajoli
> >
> >
> >
> >
> >
> > --
> > Zeno Tajoli
> > / SVILUPPO PRODOTTI/ - Automazione Biblioteche
> > Email: z.tajoli at cineca.it Fax: 051/6132198
> > *CINECA* Consorzio Interuniversitario - Sede operativa di Segrate (MI)