[Bug 13082] New: tools/export.pl fix to forbid adding of all invalid records
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13082 Bug ID: 13082 Summary: tools/export.pl fix to forbid adding of all invalid records Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Tools Assignee: gmcharlt@gmail.com Reporter: stephane.delaune@biblibre.com QA Contact: testopia@bugs.koha-community.org records exported by Koha may be invalid for many reasons such as: - records too large exceeding the 99999 bytes allowed by the leader - the encoding problems that produce a deviation between the size defined in the leader and the real size of the record when reopening (it generates many character offsets) - all other problems can generate warnings to the reopening of exported records (and the crash of most of the reimport's scripts) bz12229 only fixes the problem with records too large, but it does not exclude all the invalid records from the exported file : a more complete solution must replace it. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13082 stephane.delaune@biblibre.com changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|gmcharlt@gmail.com |stephane.delaune@biblibre.c | |om -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13082 stephane.delaune@biblibre.com changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |12229 -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13082 --- Comment #1 from stephane.delaune@biblibre.com --- Created attachment 32332 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=32332&action=edit proposed patch -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13082 stephane.delaune@biblibre.com changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13082 stephane.delaune@biblibre.com changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P5 - low |P3 Severity|enhancement |normal -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13082 stephane.delaune@biblibre.com changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #32332|0 |1 is obsolete| | --- Comment #2 from stephane.delaune@biblibre.com --- Created attachment 32557 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=32557&action=edit proposed patch proposed patch : excludes records with marc-record's errors and also other non-marc fatal errors (eg. "utf8 "\xCF" does not map to Unicode at /usr/lib/perl/5.10/Encode.pm line 174.") -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13082 Fridolin SOMERS <fridolyn.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolyn.somers@biblibre.co | |m Patch complexity|--- |Trivial patch -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13082 --- Comment #3 from Frédéric Demians <frederic@tamil.fr> --- Created attachment 32715 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=32715&action=edit Bug 13082: fix to prevent adding of invalid records in marc file Test: 1. Edit record, add 100.000 chars text to 500a 2. xml export produce the record, 3. mrc export do not produce the record, warning on log export.pl: Record length of 111000 is larger than the MARC spec allows (99999 bytes). at /usr/share/perl5/MARC/File/USMARC.pm line 314. record (number 139489) is invalid and therefore not exported because its reopening generates warnings above at... Signed-off-by: Frederic Demians <f.demians@tamil.fr> I confirm that exporting biblio records larger than 10000 characters in ISO2709 produces invalid files. After applying this patch, the culprit record (too large, but also other inconsistencies preventing record parsing with MARC::File::USMARC) is not exported anymore. A warning is produced in Koha Apache log file. Warnings to the user on WUI would be better, but it isn't the case yet, so it isn't a regression. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13082 Frédéric Demians <frederic@tamil.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off CC| |frederic@tamil.fr -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13082 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #32557|0 |1 is obsolete| | CC| |katrin.fischer@bsz-bw.de -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13082 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13082 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #32715|0 |1 is obsolete| | --- Comment #4 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 32718 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=32718&action=edit [PASSED QA] Bug 13082: fix to prevent adding of invalid records in marc file Test: 1. Edit record, add 100.000 chars text to 500a 2. xml export produce the record, 3. mrc export do not produce the record, warning on log export.pl: Record length of 111000 is larger than the MARC spec allows (99999 bytes). at /usr/share/perl5/MARC/File/USMARC.pm line 314. record (number 139489) is invalid and therefore not exported because its reopening generates warnings above at... Signed-off-by: Frederic Demians <f.demians@tamil.fr> I confirm that exporting biblio records larger than 10000 characters in ISO2709 produces invalid files. After applying this patch, the culprit record (too large, but also other inconsistencies preventing record parsing with MARC::File::USMARC) is not exported anymore. A warning is produced in Koha Apache log file. Warnings to the user on WUI would be better, but it isn't the case yet, so it isn't a regression. Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de> I agree that a visible warning/result message in the staff interface would be nice, but this works as described. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13082 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to Master CC| |tomascohen@gmail.com --- Comment #5 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Patch pushed to master. Thanks Stéphane! -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org