[Bug 35710] New: Indexing problem with "&" symbol in Zebra. Some biblio-records with "&" symbol are not searching
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35710 Bug ID: 35710 Summary: Indexing problem with "&" symbol in Zebra. Some biblio-records with "&" symbol are not searching Change sponsored?: --- Product: Koha Version: 21.05 Hardware: All OS: Linux Status: NEW Severity: normal Priority: P5 - low Component: Searching - Zebra Assignee: koha-bugs@lists.koha-community.org Reporter: yuriy.kotsyuk@oa.edu.ua I find a problem in my academy koha instance. My librarians can't find some biblio-records. I check database and figured out that records looks good. So, I decided to rebuild zebra index. During rebuilding where a lot of errors. Simple analyses show, that problem was with "&" symbol. Example below: ... Wide character in warn at /usr/share/koha/lib/C4/Biblio.pm line 1199. problem with :423324 : :42: parser error : xmlParseEntityRef: no name <subfield code="b">Norton & Company</subfield> ^ <?xml version="1.0" encoding="UTF-8"?> <record xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.loc.gov/MARC21/slim http://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd" xmlns="http://www.loc.gov/MARC21/slim"> <leader>00516nam a22002177a 4500</leader> <controlfield tag="003">UA-OsUOA</controlfield> <controlfield tag="001"> 133930 </controlfield> <controlfield tag="007">ta</controlfield> <controlfield tag="008">20181112b un ||||| |||| 00| 0 ukr d</controlfield> <datafield tag="090" ind1=" " ind2=" "> <subfield code="a">51</subfield> </datafield> <datafield tag="942" ind1=" " ind2=" "> <subfield code="c">BK</subfield> </datafield> <datafield tag="040" ind1=" " ind2=" "> <subfield code="a">UA-OsUOA</subfield> <subfield code="b">eng</subfield> <subfield code="c">UA-OsUOA</subfield> <subfield code="d">UA-OsUOA</subfield> </datafield> <datafield tag="245" ind1=" " ind2=" "> <subfield code="a">The Principles of Mathematics</subfield> <subfield code="c">B. Russell</subfield> </datafield> <datafield tag="300" ind1=" " ind2=" "> <subfield code="a">535 p. с.</subfield> </datafield> <datafield tag="955" ind1=" " ind2=" "> <subfield code="a">2</subfield> </datafield> <datafield tag="041" ind1=" " ind2=" "> <subfield code="a">eng</subfield> </datafield> <datafield tag="080" ind1=" " ind2=" "> <subfield code="a">51</subfield> </datafield> <datafield tag="260" ind1=" " ind2=" "> <subfield code="a">New York</subfield> <subfield code="b">Norton & Company</subfield> <subfield code="c">1951</subfield> </datafield> <datafield tag="852" ind1=" " ind2=" "> <subfield code="a"></subfield> </datafield> <datafield tag="650" ind1=" " ind2="7"> <subfield code="a">51 Математика</subfield> <subfield code="2">UDC</subfield> </datafield> <datafield tag="999" ind1=" " ind2=" "> <subfield code="c">423324</subfield> <subfield code="d">423324</subfield> </datafield> <datafield tag="901" ind1=" " ind2=" "> <subfield code="c">2</subfield> <subfield code="d">Юрій Анатолійович Коцюк</subfield> </datafield> </record> error retrieving biblio 423324 at /usr/share/koha/bin/migration_tools/rebuild_zebra.pl line 686. ... One more example: ... Wide character in warn at /usr/share/koha/lib/C4/Biblio.pm line 1199. problem with :423001 : :43: parser error : EntityRef: expecting ';' <subfield code="u">http://zt.knteu.kiev.ua/index.php?option=com_content&view ^ :43: parser error : EntityRef: expecting ';' ld code="u">http://zt.knteu.kiev.ua/index.php?option=com_content&view=article&id ^ :43: parser error : EntityRef: expecting ';' http://zt.knteu.kiev.ua/index.php?option=com_content&view=article&id=2349&Itemid ^ :43: parser error : EntityRef: expecting ';' .knteu.kiev.ua/index.php?option=com_content&view=article&id=2349&Itemid=721&lang ^ ... It's looks like script try use ampersand sequence... -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35710 --- Comment #1 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Hi Yuriy, 21.05 is very old and no longer maintained. I suggest that you do a test in a newer version of Koha if the problem still remains. You could use a Koha sandbox or one of the publicly available demo installations. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35710 --- Comment #2 from Yuriy Kotsyuk <yuriy.kotsyuk@oa.edu.ua> --- I can add new records without problem, even it has "&" symbol... -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35710 --- Comment #3 from Yuriy Kotsyuk <yuriy.kotsyuk@oa.edu.ua> --- Also, I found an “incorrect” record through the database and changed the “&” symbol to “ZZZ” (it’s a rare combination). Then I rebuilt the index. This record was indexed correctly and was found by the usual search. Next, I used the librarian interface to change “ZZZ” back to “&” symbol. After these operations, I got a really correct searchable bib-record. I’m not really sure that the problem is with the old koha version… -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35710 --- Comment #4 from Yuriy Kotsyuk <yuriy.kotsyuk@oa.edu.ua> --- (In reply to Katrin Fischer from comment #1)
Hi Yuriy, 21.05 is very old and no longer maintained. I suggest that you do a test in a newer version of Koha if the problem still remains. You could use a Koha sandbox or one of the publicly available demo installations.
Well, I updated the version of koha to 23.05.06.000, but the problem still exists. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35710 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au --- Comment #5 from David Cook <dcook@prosentient.com.au> --- (In reply to Yuriy Kotsyuk from comment #3)
I’m not really sure that the problem is with the old koha version…
This has the look of a data migration issue to me... When the & is added to the MARC record via the HTML form, it should be automatically converted into the entity & When I do use the HTML form, I see the following in the database: <subfield code="b">Norton & Company</subfield> If I recall correctly, this automatic entity escaping is part of the MARC::File::XML: https://metacpan.org/dist/MARC-File-XML/source/lib/MARC/File/XML.pm#L370 -- Do you know how this record was added to Koha? -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35710 --- Comment #6 from David Cook <dcook@prosentient.com.au> --- The problem is that your record isn't valid XML. While it is possible for Koha to make invalid XML via the HTML form, it's very unlikely to happen with the &. So I'd say these records are from an external source that created them without regard to XML validity. Another noticeable flaw is also the following: <datafield tag="852" ind1=" " ind2=" "> <subfield code="a"></subfield> </datafield> That would never happen in a record created by Koha I think. If there's an empty value, the subfield and field would not have been created at all. -- My suggestion would be to write a script to iterate through your records and check whether or not they can be parsed as XML. Flag the ones that cannot be parsed, and look at fixing the problems. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35710 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution|--- |INVALID --- Comment #7 from David Cook <dcook@prosentient.com.au> --- I'm going to mark this as invalid as I don't think it's a Koha bug. You can find more information on getting support with Koha here: https://koha-community.org/support/ I'm also happy to answer some follow up questions to the answers I've left here, since it seems like you're not too far away from a solution to the data problem. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35710 --- Comment #8 from Yuriy Kotsyuk <yuriy.kotsyuk@oa.edu.ua> --- Well, I see. I have to correct invalid records through direct access to database. May be one more question: where can I find log with rebuilding koha-zebra index errors? -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35710 --- Comment #9 from David Cook <dcook@prosentient.com.au> --- (In reply to Yuriy Kotsyuk from comment #8)
Well, I see. I have to correct invalid records through direct access to database. May be one more question: where can I find log with rebuilding koha-zebra index errors?
If you're running it manually, then it will just be on the screen. If it's a cronjob, then it'll be in your cron output, which will be emailed based on your system configuration. That's well outside the scope of Koha though. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org