https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43020 Bug ID: 43020 Summary: One bad record can abort an entire OAI-PMH harvest Initiative type: --- Sponsorship --- status: Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Cataloging Assignee: koha-bugs@lists.koha-community.org Reporter: martin.renvoize@openfifth.co.uk QA Contact: testopia@bugs.koha-community.org CC: m.de.rooy@rijksmuseum.nl Depends on: 35104 Target Milestone: --- Koha::OAI::Client::Harvester::processRecord has no error handling around MARC::Record->new_from_xml when decoding a harvested record's metadata, and no handling for Koha::Exceptions::Metadata::Invalid around the subsequent AddBiblio/ModBiblio/AddAuthority/ModAuthority calls either. Since init()'s harvest loop has no try/catch around processRecord, either failure dies uncaught, aborting the entire harvest_oai.pl cron run partway through - no completion log entry, no report email, and every remaining record in that batch is left unprocessed. This was found while auditing other places in the codebase that ingest externally-sourced MARC data, prompted by bug 35104 (which fixed the same class of problem - one bad record silently aborting a whole batch/job - in the staged MARC import commit/revert code paths). Test plan: 1. Configure an OAI-PMH server under Administration > OAI-PMH harvesting servers, with a record type of biblio. 2. Arrange for one harvested record to have metadata that cannot be decoded as MARCXML (e.g. an empty <datafield> element, or any malformed MARC). 3. Run: misc/cronjobs/harvest_oai.pl -r <server id> 4. Before this patch: the cron job dies partway through, with no harvest report and no indication of which record caused it. 5. After this patch: the harvest completes. The bad record is counted as in_error in the harvest report/email, its identifier and decoding error are logged, and every other record in the batch is still processed normally. 6. prove t/db_dependent/Koha/OAIHarvester.t Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35104 [Bug 35104] We should warn when attempting to save MARC records that contain characters invalid in XML -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.