[Bug 23846] New: bad iso 2709 record silently results in broken marcxml
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 Bug ID: 23846 Summary: bad iso 2709 record silently results in broken marcxml Change sponsored?: --- Product: Koha Version: 19.05 Hardware: All OS: All Status: NEW Severity: normal Priority: P3 Component: MARC Bibliographic record staging/import Assignee: koha-bugs@lists.koha-community.org Reporter: dominique@leducq.net QA Contact: testopia@bugs.koha-community.org Created attachment 94419 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=94419&action=edit iso 2709 with problematic record The (UNIMARC) ISO 2709 record in the attached file (26080509.txt) seems to have a problem : the 009 field should contain an ARK identifier but seems to be something else, with a subfield separator in it. When imported into koha, it is said to be imported all right, but when trying to access the notice, we get the following error message : Invalid data, cannot decode object (id=9865, format=marcxml, schema=UNIMARC) Investigating, I found that the marcxml in biblio_metadata.metadata is broken : the content of the <controlfield tag="009"> element is a verbatim copy of the 009 field, including the (non-printable, 0x1F) subfield separator (cf line 9 of the attached marcxml) When I deleted this element in the database, all seems to work fine. I think either the record should be rejected alltogether, with an explicit error message, or the erroneous field should be droped. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 --- Comment #1 from Dominique Leducq <dominique@leducq.net> --- Created attachment 94420 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=94420&action=edit broken marcxml record -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 Dominique Leducq <dominique@leducq.net> changed: What |Removed |Added ---------------------------------------------------------------------------- OS|All |Linux Hardware|All |PC -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 Dominique Leducq <dominique@leducq.net> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dominique@leducq.net -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |23425 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23425 [Bug 23425] Search explodes with "invalid data, cannot decode object" -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |jonathan.druart@bugs.koha-c |ity.org |ommunity.org Version|19.05 |master Status|NEW |ASSIGNED CC| |jonathan.druart@bugs.koha-c | |ommunity.org -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 --- Comment #2 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 95291 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=95291&action=edit Bug 23846: Display degraded view when MARCXML is invalid (staff detail) When an invalid bibliographic record is imported into the catalogue there is not warning or error. However the bibliographic record detail page will explode (Koha::Biblio::Metadata->record will raise an exception). This patch proposes to catch the exception on this view and display a warning about the situation. Note that editing/saving the record will fix the MARCXML data and so removes the warning (some black magic we should get rid of I suspect). Test plan: - Import a bibliographic record with invalid XML, you can add non printable characters, like 0x1F (CTRL-V 1F with vim) - Go to the detail page => Without this patch you get a 500 => With this patch applied you get a "degraded view" with a warning message, telling you what the error is. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 --- Comment #3 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 95292 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=95292&action=edit Bug 23846: Add a check to the data inconsistencies script This may be quite long for big catalogue, but I think it is a good one to have. Test plan: Same as first patch, then execute search_for_data_inconsistencies.pl Notice the error. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|bad iso 2709 record |Handle records with broken |silently results in broken |MARCXML on the |marcxml |bibliographic detail view --- Comment #4 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Bonjour Dominique, There is certainly much more to do here, but this quick fix will at least prevent the page to crash. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|normal |major -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #95291|0 |1 is obsolete| | --- Comment #5 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 95329 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=95329&action=edit Bug 23846: Display degraded view when MARCXML is invalid (staff detail) When an invalid bibliographic record is imported into the catalogue there is not warning or error. However the bibliographic record detail page will explode (Koha::Biblio::Metadata->record will raise an exception). This patch proposes to catch the exception on this view and display a warning about the situation. Note that editing/saving the record will fix the MARCXML data and so removes the warning (some black magic we should get rid of I suspect). Test plan: - Import a bibliographic record with invalid XML, you can add non printable characters, like 0x1F (CTRL-V 1F with vim) - Go to the detail page => Without this patch you get a 500 => With this patch applied you get a "degraded view" with a warning message, telling you what the error is. Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #95292|0 |1 is obsolete| | --- Comment #6 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 95330 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=95330&action=edit Bug 23846: Add a check to the data inconsistencies script This may be quite long for big catalogue, but I think it is a good one to have. Test plan: Same as first patch, then execute search_for_data_inconsistencies.pl Notice the error. Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |martin.renvoize@ptfs-europe | |.com Status|Needs Signoff |Signed Off --- Comment #7 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Works as expected, thanks Jonathan.. Signing off. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Failed QA CC| |tomascohen@gmail.com QA Contact|testopia@bugs.koha-communit |tomascohen@gmail.com |y.org | --- Comment #8 from Tomás Cohen Arazi <tomascohen@gmail.com> --- I tested the change in catalogue/detail.pl and it works as expected. I loaded the failed marcxml into a sample record and had the maintenance script display the problem, and the staff UI display it as well. I have doubts about how useful the degraded view is or in what circumstances it will display anything. I would like it to display references to the introduced maintenance script so people can take actions (for example: It is recommended that you run the following maintenance script to find (this and maybe other) problems on your data: maintenance/search_for_data_inconsistencies.pl /end_suggestion Minor issues: - Missing filter in the tt. - POD error on the script -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 --- Comment #9 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 95340 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=95340&action=edit Bug 23846: Handle exception gracefully at the OPAC I do not think we should have the same trick as the intranet, and display a message. This should be enough. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #95340|0 |1 is obsolete| | --- Comment #10 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 95341 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=95341&action=edit Bug 23846: Handle exception gracefully at the OPAC I do not think we should have the same trick as the intranet, and display a message. This should be enough. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #95329|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #95330|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #95341|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 --- Comment #11 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 95342 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=95342&action=edit Bug 23846: Display degraded view when MARCXML is invalid (staff detail) When an invalid bibliographic record is imported into the catalogue there is not warning or error. However the bibliographic record detail page will explode (Koha::Biblio::Metadata->record will raise an exception). This patch proposes to catch the exception on this view and display a warning about the situation. Note that editing/saving the record will fix the MARCXML data and so removes the warning (some black magic we should get rid of I suspect). Test plan: - Import a bibliographic record with invalid XML, you can add non printable characters, like 0x1F (CTRL-V 1F with vim) - Go to the detail page => Without this patch you get a 500 => With this patch applied you get a "degraded view" with a warning message, telling you what the error is. Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 --- Comment #12 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 95343 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=95343&action=edit Bug 23846: Add a check to the data inconsistencies script This may be quite long for big catalogue, but I think it is a good one to have. Test plan: Same as first patch, then execute search_for_data_inconsistencies.pl Notice the error. Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 --- Comment #13 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 95344 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=95344&action=edit Bug 23846: Handle exception gracefully at the OPAC I do not think we should have the same trick as the intranet, and display a message. This should be enough. Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to master Version(s)| |19.11.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 --- Comment #14 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Nice work! Pushed to master for 19.11.00 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 Fridolin SOMERS <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m Status|Pushed to master |Pushed to stable Version(s)|19.11.00 |19.11.00,19.05.05 released in| | --- Comment #15 from Fridolin SOMERS <fridolin.somers@biblibre.com> --- Pushed to 19.05.x for 19.05.05 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=24243 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=24574 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 --- Comment #16 from David Cook <dcook@prosentient.com.au> --- This is great! Thanks Jonathan, Martin, and Tomas. While I try to handle all encoding issues on the backend, it's really nice to be able to people to self-manage this. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 Fridolin SOMERS <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |26171 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26171 [Bug 26171] Show biblionumber in search_for_data_inconsistencies.pl invalid MARCXML -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |29690 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29690 [Bug 29690] Software error in details.pl when invalid MARCXML -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |29697 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29697 [Bug 29697] Excessive use of StripNonXmlChars -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |34014 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34014 [Bug 34014] There is no way to fix records with broken MARCXML -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |33270 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33270 [Bug 33270] OAI-PMH should not die on record errors -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23846 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |35099 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35099 [Bug 35099] Cannot load records with invalid marcxml -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org