[Bug 43029] New: Guard maintenance scripts against biblios with corrupt metadata
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43029 Bug ID: 43029 Summary: Guard maintenance scripts against biblios with corrupt metadata Initiative type: --- Sponsorship --- status: Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Tools Assignee: koha-bugs@lists.koha-community.org Reporter: martin.renvoize@openfifth.co.uk QA Contact: testopia@bugs.koha-community.org Depends on: 35104 Target Milestone: --- Bug 35104 makes Koha::Biblio::Metadata->record()/store() throw Koha::Exceptions::Metadata::Invalid for a record whose MARCXML cannot be decoded. misc/maintenance/sanitize_records.pl and misc/maintenance/touch_all_biblios.pl both read a biblio's metadata record and unconditionally re-save it across every biblio in the database, so a single already-corrupt record (which could exist from before bug 35104, or from a source outside Koha's own validation) would abort the whole run. This patch wraps the read (and, for touch_all_biblios.pl, the resulting ModBiblio) so one such biblio is skipped and reported rather than aborting the whole maintenance run. Depends on bug 35104. Test plan: 1. Apply this patch on top of bug 35104. 2. Create a biblio with metadata that is already corrupt, bypassing Koha::Biblio::Metadata's validation (e.g. write invalid MARCXML directly to the biblio_metadata row via the DBIC schema result, not through ->store()), alongside several normal biblios. 3. Run misc/maintenance/sanitize_records.pl --verbose across all biblios. Confirm the corrupt one is skipped with a clear "ERROR: Invalid record" message and the script completes for every other biblio without dying. 4. Run misc/maintenance/touch_all_biblios.pl similarly. Confirm the corrupt record is counted/logged as an error and the other records are still touched successfully. Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35104 [Bug 35104] We should warn when attempting to save MARC records that contain characters invalid in XML -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43029 --- Comment #1 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 201709 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201709&action=edit Bug 43029: Guard maintenance scripts against biblios with corrupt metadata sanitize_records.pl and touch_all_biblios.pl both read a biblio's metadata record and unconditionally re-save it. Since Koha::Biblio::Metadata->record()/store() can now throw Koha::Exceptions::Metadata::Invalid for a record that was already corrupt before bug 35104 existed, wrap the read (and, for touch_all_biblios.pl, the resulting ModBiblio) so one such biblio is skipped and reported rather than aborting the whole maintenance run. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43029 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #201709|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43029 --- Comment #2 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 201735 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=201735&action=edit Bug 43029: Guard maintenance scripts against biblios with corrupt metadata sanitize_records.pl and touch_all_biblios.pl both read a biblio's metadata record and unconditionally re-save it. Since Koha::Biblio::Metadata->record()/store() can now throw Koha::Exceptions::Metadata::Invalid for a record that was already corrupt before bug 35104 existed, wrap the read (and, for touch_all_biblios.pl, the resulting ModBiblio) so one such biblio is skipped and reported rather than aborting the whole maintenance run. Signed-off-by: Baptiste Bayche <baptiste.bayche@inlibro.com> Signed-off-by: Baptiste Wojtkowski <baptiste.wojtkowski@biblibre.com> Sponsored-by: OpenFifth -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43029 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43029 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43029 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |martin.renvoize@openfifth.c |ity.org |o.uk -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43029 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #201735|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43029 --- Comment #3 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 203277 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=203277&action=edit Bug 43029: Guard maintenance scripts against biblios with corrupt metadata sanitize_records.pl and touch_all_biblios.pl both read a biblio's metadata record and unconditionally re-save it. Since Koha::Biblio::Metadata->record()/store() can now throw Koha::Exceptions::Metadata::Invalid for a record that was already corrupt before bug 35104 existed, wrap the read (and, for touch_all_biblios.pl, the resulting ModBiblio) so one such biblio is skipped and reported rather than aborting the whole maintenance run. Signed-off-by: Baptiste Bayche <baptiste.bayche@inlibro.com> Signed-off-by: Baptiste Wojtkowski <baptiste.wojtkowski@biblibre.com> Sponsored-by: OpenFifth -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43029 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Comma delimited| |OpenFifth list of Sponsors| | QA Contact|testopia@bugs.koha-communit |nick@bywatersolutions.com |y.org | Patch complexity|--- |Trivial patch Sponsorship status|--- |Sponsored -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43029 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #203277|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43029 --- Comment #4 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 204744 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=204744&action=edit Bug 43029: Guard maintenance scripts against biblios with corrupt metadata sanitize_records.pl and touch_all_biblios.pl both read a biblio's metadata record and unconditionally re-save it. Since Koha::Biblio::Metadata->record()/store() can now throw Koha::Exceptions::Metadata::Invalid for a record that was already corrupt before bug 35104 existed, wrap the read (and, for touch_all_biblios.pl, the resulting ModBiblio) so one such biblio is skipped and reported rather than aborting the whole maintenance run. Signed-off-by: Baptiste Bayche <baptiste.bayche@inlibro.com> Signed-off-by: Baptiste Wojtkowski <baptiste.wojtkowski@biblibre.com> Sponsored-by: OpenFifth -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43029 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on|35104 |43777 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35104 [Bug 35104] We should warn when attempting to save MARC records that contain characters invalid in XML https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43777 [Bug 43777] Koha::Biblio::Metadata->record should repair recoverable MARCXML faults on read -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43029 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #204744|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43029 --- Comment #5 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 207722 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=207722&action=edit Bug 43029: Guard maintenance scripts against biblios with corrupt metadata sanitize_records.pl and touch_all_biblios.pl both read a biblio's metadata record and unconditionally re-save it. Since Koha::Biblio::Metadata->record()/store() can now throw Koha::Exceptions::Metadata::Invalid for a record that was already corrupt before bug 35104 existed, wrap the read (and, for touch_all_biblios.pl, the resulting ModBiblio) so one such biblio is skipped and reported rather than aborting the whole maintenance run. Signed-off-by: Baptiste Bayche <baptiste.bayche@inlibro.com> Signed-off-by: Baptiste Wojtkowski <baptiste.wojtkowski@biblibre.com> Sponsored-by: OpenFifth -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43029 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on|43777 |43025 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43025 [Bug 43025] Show metadata repair warnings in the cataloguing edit form https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43777 [Bug 43777] Koha::Biblio::Metadata->record should repair recoverable MARCXML faults on read -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43029 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #207722|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43029 --- Comment #6 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 207736 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=207736&action=edit Bug 43029: Guard maintenance scripts against biblios with corrupt metadata sanitize_records.pl and touch_all_biblios.pl both read a biblio's metadata record and unconditionally re-save it. Since Koha::Biblio::Metadata->record()/store() can now throw Koha::Exceptions::Metadata::Invalid for a record that was already corrupt before bug 35104 existed, wrap the read (and, for touch_all_biblios.pl, the resulting ModBiblio) so one such biblio is skipped and reported rather than aborting the whole maintenance run. Signed-off-by: Baptiste Bayche <baptiste.bayche@inlibro.com> Signed-off-by: Baptiste Wojtkowski <baptiste.wojtkowski@biblibre.com> Sponsored-by: OpenFifth -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43029 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #207736|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43029 --- Comment #7 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 207785 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=207785&action=edit Bug 43029: Guard maintenance scripts against biblios with corrupt metadata sanitize_records.pl and touch_all_biblios.pl both read a biblio's metadata record and unconditionally re-save it. Since Koha::Biblio::Metadata->record()/store() can now throw Koha::Exceptions::Metadata::Invalid for a record that was already corrupt before bug 35104 existed, wrap the read (and, for touch_all_biblios.pl, the resulting ModBiblio) so one such biblio is skipped and reported rather than aborting the whole maintenance run. Signed-off-by: Baptiste Bayche <baptiste.bayche@inlibro.com> Signed-off-by: Baptiste Wojtkowski <baptiste.wojtkowski@biblibre.com> Sponsored-by: OpenFifth -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43029 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43029 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #207785|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43029 --- Comment #8 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 209858 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=209858&action=edit Bug 43029: Guard maintenance scripts against biblios with corrupt metadata sanitize_records.pl and touch_all_biblios.pl both read a biblio's metadata record and unconditionally re-save it. Since Koha::Biblio::Metadata->record()/store() can now throw Koha::Exceptions::Metadata::Invalid for a record that was already corrupt before bug 35104 existed, wrap the read (and, for touch_all_biblios.pl, the resulting ModBiblio) so one such biblio is skipped and reported rather than aborting the whole maintenance run. Signed-off-by: Baptiste Bayche <baptiste.bayche@inlibro.com> Signed-off-by: Baptiste Wojtkowski <baptiste.wojtkowski@biblibre.com> Sponsored-by: OpenFifth Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43029 --- Comment #9 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Invalid characters and empty subfields both work fine, to see this effect you have to truly break a record, and that's easy enough: UPDATE biblio_metadata SET metadata = REPLACE(metadata, '</record>', '<datafield tag="500" ind1=" " ind2=" "></datafield><datafield tag="501" ind1=" " ind2=" "><subfield code="a">This &anp; that</subfield></datafield></record>') WHERE biblionumber =228; Note that that breakage prevents you from loading the record in the staff interface - we probably need to build in a protection for this into the cleaning routine -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org