Re: [Koha-devel] Zebra config problem (still 1)

8 Feb 2006

      Paul,

please send your marcxml record file + zebra.cfg + collection.abs . I 
might be able to see what's wrong, then

/ Adam

Paul POULAIN wrote:
...
Adam Dickmeiss a écrit :
(answer to Adam question at the end)
I want to completly describe my history with zebra, to let you be aware 
of all I did, and maybe understand why I begin to really feel 
*discouraged* :
* just in case you don't know : i've been Koha Release Manager for 
version 2.0 and 2.2. I'm the main -almost only- author of the MARC 
support in Koha.
* when the 3.0 Release Manager was nominated, Joshua, he suggested to 
adopt Zebra. At first, I was not very happy with this proposal, as it 
adds a new tool for Koha, and makes install more complex. But other args 
convinced me it was the way to go.
* Thus I set up zebra on my computer, and began to move MARC stuff to 
zebra. I succedeed to have something working correctly after something 
like a week of work. The problem being that the zebra indexing was done 
through a perl exec() and zebraidx.
So, I waited for Perl-ZOOM very impatiently, letting the code as it for 
some months (2-3 ?).
When Perl-ZOOM arrived, I was very very happy.
But now i'm really no more happy at all, as I ran into many many many 
problems and feel quite stuck and alone with the problem.
I don't want to count how many days I've spend on koha/zebra without 
success, but that's something like 6-7 full days, probably more :-(
Here is a summary of all my problems :
* at 1st, I tried to setup a iso2709 (full MARC) DB. I ran into "Error 
updating 10002 => Encoding failed". After investigating and asking this 
list, 
(http://lists.gnu.org/archive/html/koha-zebra/2006-01/msg00015.html and 
following thread) it appears that iso2709 support was problematic and 
that I had better going XML.
That seemed a good idea to me, as XML is highly more comprehensive and a 
sex-appealing technology ;-)
* Thus, I changes some code in Koha to use MARCXML package 
(http://search.cpan.org/~esummers/MARC-XML-0.81/lib/MARC/File/XML.pm)
* But I still ran into the "Error updating 10002" After investigating a 
little bit more, adam finaly caught the culprit 
(http://lists.gnu.org/archive/html/koha-zebra/2006-01/msg00034.html). 
This time it was a compilation problem !!!
* Could it be my last problem ? no, unfortunatly. I ran into the 2 
recent problems : impossible to search, failure to index with RecordId.
* It appears finally to Mike 
(http://lists.gnu.org/archive/html/koha-zebra/2006-01/msg00038.html) 
that the search features were not in official yaz package, and a new 
package has been released !
* I'm still stuck with the indexing problem. I really thought I wanted 
to do something simple : index MARCXML data (produced by ed package) 
into zebra. Why it does not work is NOT clear to me.
I solved a problem with marc21.abs to be renamed to collection.abs, but 
didn't saw anything on this, and if Tümer had not seen this, I would not 
have found it myself ! (and i't still unclear to me why you have a 
marc21.abs where MACXML speaks of <collection> tag)
Now,I'm afraid there's still something undocumented somewhere, or 
bugged, or unreleased, or something like this.
I really begin to feel discouraged and alone.
Many thanks to Tümer that pointed me some problems, but seems as stuck 
as me :-(
I end with an answer to Adam suggestion with zebraidx -s update 
testrec.xml :
...
...
<?xml version="1.0" encoding="UTF-8"?>
<collection xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.loc.gov/MARC21/slim http://www.l ...
Your root element is collection. Not record. I don't think melm will
match that. Had you used record as root element - it should do it.
It's always a good idea to try things out with
  zebraidx -s update testrec.xml
and see what gets matched.. (Look for the Idx: lines).
for XML :
...
<?xml version="1.0" encoding="UTF-8"?>
<collection xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
xsi:schemaLocation="http://www.loc.gov/MARC21/slim 
http://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd" 
xmlns="http://www.loc.gov/MARC21/slim">   
<record>
    <leader>00543     2200181   4500</leader>
    <controlfield tag="001">19</controlfield>
    <datafield tag="010" ind1=" " ind2=" ">
        <subfield code="a">2010140001</subfield>
        <subfield code="d">45 F</subfield>
    </datafield>
    <datafield tag="090" ind1=" " ind2=" ">
        <subfield code="9">16</subfield>
        <subfield code="a">16</subfield>
    </datafield>
    <datafield tag="100" ind1=" " ind2=" ">
        <subfield code="a">1995                y0fre 0103    
ba</subfield>
    </datafield>
    <datafield tag="101" ind1=" " ind2=" ">
        <subfield code="a">fre</subfield>
    </datafield>
    <datafield tag="105" ind1=" " ind2=" ">
        <subfield code="a">y       00  y</subfield>
    </datafield>
    <datafield tag="200" ind1="1" ind2=" ">
        <subfield code="a">Pour l'honneur de l'esprit humain</subfield>
        <subfield code="b">LIVR</subfield>
        <subfield code="e">Les mathematiques aujourd'hui</subfield>
        <subfield code="f">Jean DIEUDONNE</subfield>
    </datafield>
    <datafield tag="995" ind1=" " ind2=" ">
        <subfield code="b">CDI</subfield>
        <subfield code="c">CDI</subfield>
        <subfield code="e">SL</subfield>
        <subfield code="f">Non inventorie</subfield>
        <subfield code="j">000006</subfield>
        <subfield code="o">2</subfield>
        <subfield code="9">27</subfield>
    </datafield>
</record>
</collection>
...
Record type: 'collection'
    Local tag: 'collection'
         tag=collection/
                Local tag: 'subfield'
                     tag=subfield/datafield/record/collection/
                    Data: '16'
              Idx: [w]bib1:Identifier-standard [1007] data XData:"16"
              Idx: [p]bib1:Identifier-standard [1007] data XData:"16"
                        Idx: [w]bib1:Any [1016] data XData:"16"
                     tag=subfield/datafield/record/collection/
                Data: '
                '
            Local tag: 'datafield'
                 tag=datafield/record/collection/
                Data: '
                        '
                Local tag: 'subfield'
                     tag=subfield/datafield/record/collection/
                    Data: 'Pour l'honneur de l'esprit humain'
Idx: [w]bib1:Title [4] data XData:"Pour l'honneur de l'esprit humain"
Idx: [p]bib1:Title [4] data XData:"Pour l'honneur de l'esprit humain"
Idx: [w]bib1:Any [1016] data XData:"Pour l'honneur de l'esprit humain"
                     tag=subfield/datafield/record/collection/
                Data: '
                        '
11:31:48-08/02 zebraidx(26418) [log] zebra_end_trans
11:31:48-08/02 zebraidx(26418) [log] sorting section 1
11:31:48-08/02 zebraidx(26418) [log] Iterations . . .     42
11:31:48-08/02 zebraidx(26418) [log] Distinct words .     20
11:31:48-08/02 zebraidx(26418) [log] Updates. . . . .     17
11:31:48-08/02 zebraidx(26418) [log] Deletions. . . .      1
11:31:48-08/02 zebraidx(26418) [log] Insertions . . .      2
11:31:48-08/02 zebraidx(26418) [log][app2] zebra_register_close
with zebraidx -s update testrec.xml I get (many lines snipped, complete 
log at end of mail) :
p=0x8106c70
...
11:31:48-08/02 zebraidx(26418) [log] Records:       0 i/u/d 0/0/0
11:31:48-08/02 zebraidx(26418) [log] user/system: 0/0
11:31:48-08/02 zebraidx(26418) [log][app2] zebra_stop
11:31:48-08/02 zebraidx(26418) [log] zebraidx times:  0.06  0.00  0.00
[paul@bureau unimarc]$
If I read correctly, The Identifier-standard [1007] is correctly 
detected, but it does not work anymore.
The complete log from zebraidx :
==========================================================
...
Record type: 'collection'
    Local tag: 'collection'
         tag=collection/
        Data: '
'
        Local tag: 'record'
             tag=record/collection/
            Data: '
                '
            Local tag: 'leader'
                 tag=leader/record/collection/
                Data: '00543     2200181   4500'
                 tag=leader/record/collection/
            Data: '
                '
            Local tag: 'controlfield'
                 tag=controlfield/record/collection/
                Data: '19'
                 tag=controlfield/record/collection/
            Data: '
                '
            Local tag: 'datafield'
                 tag=datafield/record/collection/
                Data: '
                        '
                Local tag: 'subfield'
                     tag=subfield/datafield/record/collection/
                    Data: '2010140001'
                     tag=subfield/datafield/record/collection/
                Data: '
                        '
                Local tag: 'subfield'
                     tag=subfield/datafield/record/collection/
                    Data: '45 F'
                     tag=subfield/datafield/record/collection/
                Data: '
                '
                 tag=datafield/record/collection/
            Data: '
                '
            Local tag: 'datafield'
                 tag=datafield/record/collection/
                Data: '
                        '
                Local tag: 'subfield'
                     tag=subfield/datafield/record/collection/
                    Data: '16'
                     tag=subfield/datafield/record/collection/
                Data: '
                        '
                Local tag: 'subfield'
                     tag=subfield/datafield/record/collection/
                    Data: '16'
                        Idx: [w]bib1:Identifier-standard [1007] data 
XData:"16"
                        Idx: [p]bib1:Identifier-standard [1007] data 
XData:"16"
                        Idx: [w]bib1:Any [1016] data XData:"16"
                     tag=subfield/datafield/record/collection/
                Data: '
                '
                 tag=datafield/record/collection/
            Data: '
                '
            Local tag: 'datafield'
                 tag=datafield/record/collection/
                Data: '
                        '
                Local tag: 'subfield'
                     tag=subfield/datafield/record/collection/
                    Data: '1995                y0fre 0103    ba'
                     tag=subfield/datafield/record/collection/
                Data: '
                '
                 tag=datafield/record/collection/
            Data: '
                '
            Local tag: 'datafield'
                 tag=datafield/record/collection/
                Data: '
                        '
                Local tag: 'subfield'
                     tag=subfield/datafield/record/collection/
                    Data: 'fre'
                     tag=subfield/datafield/record/collection/
                Data: '
                '
                 tag=datafield/record/collection/
            Data: '
                '
            Local tag: 'datafield'
                 tag=datafield/record/collection/
                Data: '
                        '
                Local tag: 'subfield'
                     tag=subfield/datafield/record/collection/
                    Data: 'y       00  y'
                     tag=subfield/datafield/record/collection/
                Data: '
                '
                 tag=datafield/record/collection/
            Data: '
                '
            Local tag: 'datafield'
                 tag=datafield/record/collection/
                Data: '
                        '
                Local tag: 'subfield'
                     tag=subfield/datafield/record/collection/
                    Data: 'Pour l'honneur de l'esprit humain'
                        Idx: [w]bib1:Title [4] data XData:"Pour 
l'honneur de l'esprit humain"
                        Idx: [p]bib1:Title [4] data XData:"Pour 
l'honneur de l'esprit humain"
                        Idx: [w]bib1:Any [1016] data XData:"Pour 
l'honneur de l'esprit humain"
                     tag=subfield/datafield/record/collection/
                Data: '
                        '
                Local tag: 'subfield'
                     tag=subfield/datafield/record/collection/
                    Data: 'LIVR'
                     tag=subfield/datafield/record/collection/
                Data: '
                        '
                Local tag: 'subfield'
                     tag=subfield/datafield/record/collection/
                    Data: 'Les mathematiques aujourd'hui'
                        Idx: [w]bib1:Title [4] data XData:"Les 
mathematiques aujourd'hui"
                        Idx: [p]bib1:Title [4] data XData:"Les 
mathematiques aujourd'hui"
                        Idx: [w]bib1:Any [1016] data XData:"Les 
mathematiques aujourd'hui"
                     tag=subfield/datafield/record/collection/
                Data: '
                        '
                Local tag: 'subfield'
                     tag=subfield/datafield/record/collection/
                    Data: 'Jean DIEUDONNE'
                     tag=subfield/datafield/record/collection/
                Data: '
                '
                 tag=datafield/record/collection/
            Data: '
                '
            Local tag: 'datafield'
                 tag=datafield/record/collection/
                Data: '
                        '
                Local tag: 'subfield'
                     tag=subfield/datafield/record/collection/
                    Data: 'CDI'
                     tag=subfield/datafield/record/collection/
                Data: '
                        '
                Local tag: 'subfield'
                     tag=subfield/datafield/record/collection/
                    Data: 'CDI'
                     tag=subfield/datafield/record/collection/
                Data: '
                        '
                Local tag: 'subfield'
                     tag=subfield/datafield/record/collection/
                    Data: 'SL'
                     tag=subfield/datafield/record/collection/
                Data: '
                        '
                Local tag: 'subfield'
                     tag=subfield/datafield/record/collection/
                    Data: 'Non inventorie'
                     tag=subfield/datafield/record/collection/
                Data: '
                        '
                Local tag: 'subfield'
                     tag=subfield/datafield/record/collection/
                    Data: '000006'
                     tag=subfield/datafield/record/collection/
                Data: '
                        '
                Local tag: 'subfield'
                     tag=subfield/datafield/record/collection/
                    Data: '2'
                     tag=subfield/datafield/record/collection/
                Data: '
                        '
                Local tag: 'subfield'
                     tag=subfield/datafield/record/collection/
                    Data: '27'
                     tag=subfield/datafield/record/collection/
                Data: '
                '
                 tag=datafield/record/collection/
            Data: '
        '
             tag=record/collection/
        Data: '
'
         tag=collection/
-------------
11:31:48-08/02 zebraidx(26418) [log] zebra_end_trans
11:31:48-08/02 zebraidx(26418) [log] sorting section 1
11:31:48-08/02 zebraidx(26418) [log] Iterations . . .     42
11:31:48-08/02 zebraidx(26418) [log] Distinct words .     20
11:31:48-08/02 zebraidx(26418) [log] Updates. . . . .     17
11:31:48-08/02 zebraidx(26418) [log] Deletions. . . .      1
11:31:48-08/02 zebraidx(26418) [log] Insertions . . .      2
11:31:48-08/02 zebraidx(26418) [log][app2] zebra_register_close 
p=0x8106c70
11:31:48-08/02 zebraidx(26418) [log] Records:       0 i/u/d 0/0/0
11:31:48-08/02 zebraidx(26418) [log] user/system: 0/0
11:31:48-08/02 zebraidx(26418) [log][app2] zebra_stop
11:31:48-08/02 zebraidx(26418) [log] zebraidx times:  0.06  0.00  0.00
[paul@bureau 
unimarc]$