[Bug 9496] New: rebuild_zebra.pl should handle invalid XML records better
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9496 Bug ID: 9496 Summary: rebuild_zebra.pl should handle invalid XML records better Classification: Unclassified Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Command-line Utilities Assignee: gmcharlt@gmail.com Reporter: gmcharlt@gmail.com If a bib or authority MARCXML record is not well-formed, rebuild_zebra.pl should catch it so that invalid XML is not included in the file it passes to zebraidx. This is particularly important when DOM mode indexing is used. This is because in DOM mode zebraidx is processing a single XML document (namely, the MARCXML import), and if it reaches a record that causes the XML file to be invalid, it will not handle any records past that point. Worse, it doesn't complain quite loudly enough when that happens, so a user could find their Koha catalog to be partially indexed for no apparent reason. In addition, when rebuild_zebra is run using the -nosanitize option, zebraidx in DOM mode will refuse to process the output because there are extraneous XML declarations. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9496 Galen Charlton <gmcharlt@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9496 --- Comment #1 from Galen Charlton <gmcharlt@gmail.com> --- Created attachment 14869 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=14869&action=edit bug 9496: improve error checking in rebuild_zebra.pl When using rebuild_zebra to index all records, skip over bibliographic or authority records that don't come out as valid XML. Also, strip extraneous XML declarations when using --nosanitize. Test plans ---------- Note that both plans assume that DOM indexing is turned on. Test plan #1 ============ [1] Run rebuild_zebra.pl with the -x -nosanitize options. Without the patch, zebraidx should terminate early and complain about invalid XML. [2] With the patch, the rebuild_zebra.pl should work without error. Test plan #2 ============ [1] Intentionally make a MARCXML record invalid, e.g, by running the following SQL: UPDATE bilbioitems SET marcxml = CONCATENATE(marcxml, 'junk') WHERE biblionumber = 123; [2] Run rebuild_zebra.pl -b -x -r [3] Without the patch, only part of the database will be indexed. [4] With the patch, rebuild_zebra.pl will not export the bad record and will give an error message saying so, but will successfully index the rest of the records. Signed-off-by: Galen Charlton <gmc@esilibrary.com> -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9496 Larry Baerveldt <larry@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off CC| |larry@bywatersolutions.com --- Comment #2 from Larry Baerveldt <larry@bywatersolutions.com> --- Patch passed testing. Signing off. --Larry Baerveldt -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9496 Larry Baerveldt <larry@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #14869|0 |1 is obsolete| | --- Comment #3 from Larry Baerveldt <larry@bywatersolutions.com> --- Created attachment 14898 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=14898&action=edit Signed off patch bug 9496 -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9496 Mason James <mtj@kohaaloha.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9496 --- Comment #4 from Mason James <mtj@kohaaloha.com> --- Created attachment 16444 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=16444&action=edit bug 9496: improve error checking in rebuild_zebra.pl When using rebuild_zebra to index all records, skip over bibliographic or authority records that don't come out as valid XML. Also, strip extraneous XML declarations when using --nosanitize. Test plans ---------- Note that both plans assume that DOM indexing is turned on. Test plan #1 ============ [1] Run rebuild_zebra.pl with the -x -nosanitize options. Without the patch, zebraidx should terminate early and complain about invalid XML. [2] With the patch, the rebuild_zebra.pl should work without error. Test plan #2 ============ [1] Intentionally make a MARCXML record invalid, e.g, by running the following SQL: UPDATE bilbioitems SET marcxml = CONCATENATE(marcxml, 'junk') WHERE biblionumber = 123; [2] Run rebuild_zebra.pl -b -x -r [3] Without the patch, only part of the database will be indexed. [4] With the patch, rebuild_zebra.pl will not export the bad record and will give an error message saying so, but will successfully index the rest of the records. Signed-off-by: Galen Charlton <gmc@esilibrary.com> Signed-off-by: Larry Baerveldt <larry@bywatersolutions.com> Signed-off-by: Mason James <mtj@kohaaloha.com> -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9496 Mason James <mtj@kohaaloha.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mtj@kohaaloha.com --- Comment #5 from Mason James <mtj@kohaaloha.com> ---
Signed-off-by: Mason James <mtj@kohaaloha.com>
passing QA, patch tested and works well testing 1 commit(s) (applied to 2f3cf47 'Merge branch 'bug_8979' into 3.12-mas') -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9496 Mason James <mtj@kohaaloha.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #14898|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9496 Paul Poulain <paul.poulain@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #16444|0 |1 is obsolete| | --- Comment #6 from Paul Poulain <paul.poulain@biblibre.com> --- Created attachment 16514 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=16514&action=edit [PASSED QA] bug 9496: improve error checking in rebuild_zebra.pl When using rebuild_zebra to index all records, skip over bibliographic or authority records that don't come out as valid XML. Also, strip extraneous XML declarations when using --nosanitize. Test plans ---------- Note that both plans assume that DOM indexing is turned on. Test plan #1 ============ [1] Run rebuild_zebra.pl with the -x -nosanitize options. Without the patch, zebraidx should terminate early and complain about invalid XML. [2] With the patch, the rebuild_zebra.pl should work without error. Test plan #2 ============ [1] Intentionally make a MARCXML record invalid, e.g, by running the following SQL: UPDATE bilbioitems SET marcxml = CONCATENATE(marcxml, 'junk') WHERE biblionumber = 123; [2] Run rebuild_zebra.pl -b -x -r [3] Without the patch, only part of the database will be indexed. [4] With the patch, rebuild_zebra.pl will not export the bad record and will give an error message saying so, but will successfully index the rest of the records. Signed-off-by: Galen Charlton <gmc@esilibrary.com> Signed-off-by: Larry Baerveldt <larry@bywatersolutions.com> Signed-off-by: Mason James <mtj@kohaaloha.com> Signed-off-by: Paul Poulain <paul.poulain@biblibre.com> -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9496 --- Comment #7 from Paul Poulain <paul.poulain@biblibre.com> --- QA comment: * small patch, passes koha-qa.pl * i've checked it does not break GRS indexeing * looks good & sound logical * I don't have a DOM test database, so I can't test this patch working as advertised BUT : patch waiting for more than 2 months, and trusting author, so passed QA -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9496 Paul Poulain <paul.poulain@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |paul.poulain@biblibre.com QA Contact| |paul.poulain@biblibre.com --- Comment #8 from Paul Poulain <paul.poulain@biblibre.com> --- mason, seems we passed QA this patch almost at the same time ! -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9496 Jared Camins-Esakov <jcamins@cpbibliography.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to Master --- Comment #9 from Jared Camins-Esakov <jcamins@cpbibliography.com> --- This patch has been pushed to master. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9496 Chris Cormack <chris@bigballofwax.co.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to Master |Pushed to Stable CC| |chris@bigballofwax.co.nz --- Comment #10 from Chris Cormack <chris@bigballofwax.co.nz> --- Pushed to 3.10.x will be in 3.10.5 -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org