[Bug 42528] New: metadata_record should always strip nonxml characters
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42528 Bug ID: 42528 Summary: metadata_record should always strip nonxml characters Initiative type: --- Sponsorship --- status: Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Architecture, internals, and plumbing Assignee: koha-bugs@lists.koha-community.org Reporter: nick@bywatersolutions.com QA Contact: testopia@bugs.koha-community.org This was inspired by noticing that when importing a batch the indexing could fail for a batch of records where a single record had encoding issues. Koha::Biblio->metdata_record is used for retrieving records from the DB, and possibly filtering them. In the cases we are using this, it's better to give the user/caller a result than nothing The error on the record is logged, it can be seen when viewing the record in the staff interface, and can be found using the data inconsistencies tool. I think it makes sense for the default behavior to attempt to clean the record. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42528 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff Patch complexity|--- |Trivial patch -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42528 --- Comment #1 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 198613 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=198613&action=edit Bug 42528: Make metadata_record use record_strip_nonxml by default This patch adds protection to fetching the record by default to allow users and internal callers to proceed without errors when a record has minor encoding issues This also makes 'get_coins' use 'metadata_record' rather than 'metdata->record' but this doesn't have additional affects To test: 1 - View record 369 in the staff interface: http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=369 2 - Note warnings about encoding errors 3 - Try to click the 'MARC' or 'ISBD' tabs, try to export the record 4 - Fail with error: Invalid data, cannot decode metadata object (biblio_metadata.id=368, biblionumber=369, format=marcxml, schema=MARC21, decoding_error=':8: parser error : PCDATA invalid Char value 31 <controlfield tag="001">00␟aD000015937</controlfield> 5 - Search for 'Strk̆v̜elser' - confirm no results 6 - Reindex records sudo koha-elasticsearch --rebuild kohadev sudo koha-rebuild-zebra -v -f kohadev 7 - Note the errors 8 - Apply patch, restart all 9 - Reload details page and confirm you can view all the pages, and export 10 - reindex all records as above 11 - Confirm a search works -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42528 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |nick@bywatersolutions.com |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42528 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42528 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #198613|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42528 --- Comment #2 from David Nind <david@davidnind.com> --- Created attachment 198788 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=198788&action=edit Bug 42528: Make metadata_record use record_strip_nonxml by default This patch adds protection to fetching the record by default to allow users and internal callers to proceed without errors when a record has minor encoding issues This also makes 'get_coins' use 'metadata_record' rather than 'metdata->record' but this doesn't have additional affects To test: 1 - View record 369 in the staff interface: http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=369 2 - Note warnings about encoding errors 3 - Try to click the 'MARC' or 'ISBD' tabs, try to export the record 4 - Fail with error: Invalid data, cannot decode metadata object (biblio_metadata.id=368, biblionumber=369, format=marcxml, schema=MARC21, decoding_error=':8: parser error : PCDATA invalid Char value 31 <controlfield tag="001">00␟aD000015937</controlfield> 5 - Search for 'Strk̆v̜elser' - confirm no results 6 - Reindex records sudo koha-elasticsearch --rebuild kohadev sudo koha-rebuild-zebra -v -f kohadev 7 - Note the errors 8 - Apply patch, restart all 9 - Reload details page and confirm you can view all the pages, and export 10 - reindex all records as above 11 - Confirm a search works Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42528 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |martin.renvoize@openfifth.c | |o.uk See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=35104 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42528 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |martin.renvoize@openfifth.c |y.org |o.uk -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42528 --- Comment #3 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- I really think we should be focusing on bug 35104 to prevent errant records getting into the database in the first place, personally. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42528 Jonathan Druart <jonathan.druart@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@gmail.com Status|Signed Off |Failed QA --- Comment #4 from Jonathan Druart <jonathan.druart@gmail.com> --- I don't think this is what we want. I agree with Martin, we should fix before bad data get into the DB. Hiding the problem is not a good idea IMO. Failing QA because we need unit test anyway, and I guess we could remove code in other places if we finally decide to hide the errors. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42528 --- Comment #5 from Jonathan Druart <jonathan.druart@gmail.com> --- After testing: the error is actually still there. However /cgi-bin/koha/catalogue/moredetail.pl?biblionumber=369 is still dying (badly) Shouldn't we enforce the try-catch pattern in the different controllers instead? -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org