[Bug 18757] New: Problem when importing only items in MARC records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18757 Bug ID: 18757 Summary: Problem when importing only items in MARC records Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Tools Assignee: gmcharlt@gmail.com Reporter: fridolin.somers@biblibre.com QA Contact: testopia@bugs.koha-community.org When importing records with Stage MARC records for import, one can use matching rules to only import items into existing records. Those imported items are stored as XML to be staged. The bug is that when MARC Flavour is UNIMARC the XML serialization fails because its is looking in field 100$a which does not exist. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18757 Fridolin SOMERS <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|gmcharlt@gmail.com |fridolin.somers@biblibre.co | |m Severity|enhancement |normal -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18757 Fridolin SOMERS <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff Patch complexity|--- |Trivial patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18757 --- Comment #1 from Fridolin SOMERS <fridolin.somers@biblibre.com> --- Created attachment 64104 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=64104&action=edit Bug 18757 - Problem when importing only items in MARC records When importing records with Stage MARC records for import, one can use matching rules to only import items into existing records. Those imported items are stored as XML to be staged. The bug is that when MARC Flavour is UNIMARC the XML serialization fails because its is looking in field 100$a which does not exist. You see in logs the error : Unsupported UNIMARC character encoding [] for XML output for UNIMARC; 100$a This patch adds the format "USMARC" to XML serialization, like in C4::Items::_get_unlinked_subfields_xml Test plan : - On a UNIMARC database - Define a maching rule on title 200$a - Select a record with items - Export it using : Save as > MARC (Unicode/UTF-8) - Delete all items - Go to Tools > Stage MARC records for import - Upload exported file - Select title matching rule - Select "Ingore incoming record" in "Action if matching record found :" - Select Yes and "Always add items" in "Check for embedded item record data?" - Click Stage for import => Without patch you get the error => With patch the import is staged - Import into the catalog and check item is well recreated -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18757 Alex Buckley <alexbuckley@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |alexbuckley@catalyst.net.nz Status|Needs Signoff |In Discussion --- Comment #2 from Alex Buckley <alexbuckley@catalyst.net.nz> --- Hi Fridolin When applying your patch on a clean branch up to date with master, I find that I am still getting the error message 'Unsupported UNIMARC character encoding [] for XML output for UNIMARC; 100$a' as I do without the patch applied. I am getting this error when trying to both create and import records. I have tried setting the matching record rule to both 100$a and 200$a but continue to get the same error. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18757 --- Comment #3 from Fridolin SOMERS <fridolin.somers@biblibre.com> --- Looks like this only occurs with Plack. One must test with plack on tools/manage-marc-import.pl. Since there are problems with background jobs with plack, test with Bug 17240. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18757 Fridolin SOMERS <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=17240 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18757 --- Comment #4 from Fridolin SOMERS <fridolin.somers@biblibre.com> --- But this patch is I think logic since it does XML serialization like in C4::Items::_get_unlinked_subfields_xml -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18757 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |tomascohen@gmail.com --- Comment #5 from Tomás Cohen Arazi <tomascohen@gmail.com> --- (In reply to Alex Buckley from comment #2)
Hi Fridolin
When applying your patch on a clean branch up to date with master, I find that I am still getting the error message 'Unsupported UNIMARC character encoding [] for XML output for UNIMARC; 100$a' as I do without the patch applied.
I am getting this error when trying to both create and import records.
I have tried setting the matching record rule to both 100$a and 200$a but continue to get the same error.
Make sure you restart Plack after applying the patch. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18757 Fridolin SOMERS <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Discussion |Needs Signoff --- Comment #6 from Fridolin SOMERS <fridolin.somers@biblibre.com> --- I set back to NSO -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18757 M. Tompsett <mtompset@hotmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mtompset@hotmail.com --- Comment #7 from M. Tompsett <mtompset@hotmail.com> --- https://archive.ifla.org/VI/8/projects/UNIMARC-HoldingsFormat.pdf -- but isn't there supposed to be a 100?! 12-15 Character Set (Mandatory) These four character positions indicate the principal graphic character sets used in the record. Positions 12-13 designate the G0 set and positions 14-15 designate the G1 set. If a G1 set is not needed, positions 14-15 contain blanks. 01 = ISO 646, IRV version (basic Latin set) 02 = ISO Registration # 37 (basic Cyrillic set) 03 = ISO 5426 (extended Latin set) 04 = ISO DIS 5427 (extended Cyrillic set) 05 = ISO 5428 (Greek set) 06 = ISO 6438 (African coded character set) 07 = ISO 10586 (Georgian set) 08 = ISO 8957 (Hebrew set) Table 1 09 = ISO 8957 (Hebrew set) Table 2 10 = [Reserved] 11 = ISO 5426-2 (Latin characters used in minor European languages and obsolete typography) 50 = ISO 10646 Level 3 (Unicode) Oh yuck... reading the MARC::File::XML library... it basically fakes it. # Does the record think it is already Unicode? if ($original_encoding ne 'a' && lc($format) !~ /^unimarc/o) { # If not, we'll make it so $_transcode++; substr($ldr,9,1,'a'); $record->leader( $ldr ); } which is why the as_xml in elsewhere does the "USMARC" thing. Okay. I'll test this, since I'm assuming Fridolin has already tested in a UNIMARC context before submitting this patch. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18757 --- Comment #8 from M. Tompsett <mtompset@hotmail.com> --- BTW, it should be noted that this same problem shows itself as an "Internal Server Error" if one tries to manually catalogue a record in UNIMARC, and uses something clearly bad, like "KABOOM!" in the 100$a field. :) substr outside of string at /usr/share/perl5/MARC/File/XML.pm line 561 (#2) (W substr)(F) You tried to reference a substr() that pointed outside of a string. That is, the absolute value of the offset was larger than the length of the string. See "substr" in perlfunc. This warning is fatal if substr is used in an lvalue context (as the left hand side of an assignment or as a subroutine argument for example). Use of uninitialized value $enc in string eq at /usr/share/perl5/MARC/File/XML.pm line 563 (#1) Use of uninitialized value $enc in string eq at /usr/share/perl5/MARC/File/XML.pm line 565 (#1) Use of uninitialized value $enc in concatenation (.) or string at /usr/share/perl5/MARC/File/XML.pm line 568 (#1) Unsupported UNIMARC character encoding [] for XML output for UNIMARC; 100$a -> kaboom at /usr/share/perl5/MARC/File/XML.pm line 568. No need to make any 200 rules. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18757 --- Comment #9 from Fridolin SOMERS <fridolin.somers@biblibre.com> --- (In reply to M. Tompsett from comment #7)
which is why the as_xml in elsewhere does the "USMARC" thing. Okay. I'll test this, since I'm assuming Fridolin has already tested in a UNIMARC context before submitting this patch.
Indeed. Several of our Koha UNIMARC have already this patch. UNIMARC 100$a is a spaghetti dish ;-) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18757 Serhij Dubyk <dubyk@ukr.net> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dubyk@ukr.net -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18757 José-Mario Monteiro-Santos <jose-mario.monteiro-santos@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jose-mario.monteiro-santos@ | |inlibro.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18757 José-Mario Monteiro-Santos <jose-mario.monteiro-santos@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC|jose-mario.monteiro-santos@ | |inlibro.com | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18757 Giuseppe Ciaccio <giuseppe.ciaccio@unige.it> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |giuseppe.ciaccio@unige.it -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18757 sandboxes@biblibre.com <sandboxes@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off CC| |sandboxes@biblibre.com --- Comment #10 from sandboxes@biblibre.com <sandboxes@biblibre.com> --- Patch tested with a sandbox, by Amandine Zocca <azocca@ville-montauban.fr> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18757 sandboxes@biblibre.com <sandboxes@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #64104|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18757 --- Comment #11 from sandboxes@biblibre.com <sandboxes@biblibre.com> --- Created attachment 93363 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=93363&action=edit Bug 18757 - Problem when importing only items in MARC records When importing records with Stage MARC records for import, one can use matching rules to only import items into existing records. Those imported items are stored as XML to be staged. The bug is that when MARC Flavour is UNIMARC the XML serialization fails because its is looking in field 100$a which does not exist. You see in logs the error : Unsupported UNIMARC character encoding [] for XML output for UNIMARC; 100$a This patch adds the format "USMARC" to XML serialization, like in C4::Items::_get_unlinked_subfields_xml Test plan : - On a UNIMARC database - Define a maching rule on title 200$a - Select a record with items - Export it using : Save as > MARC (Unicode/UTF-8) - Delete all items - Go to Tools > Stage MARC records for import - Upload exported file - Select title matching rule - Select "Ingore incoming record" in "Action if matching record found :" - Select Yes and "Always add items" in "Check for embedded item record data?" - Click Stage for import => Without patch you get the error => With patch the import is staged - Import into the catalog and check item is well recreated Signed-off-by: Amandine Zocca <azocca@ville-montauban.fr> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18757 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #93363|0 |1 is obsolete| | --- Comment #12 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 94826 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=94826&action=edit Bug 18757 - Problem when importing only items in MARC records When importing records with Stage MARC records for import, one can use matching rules to only import items into existing records. Those imported items are stored as XML to be staged. The bug is that when MARC Flavour is UNIMARC the XML serialization fails because its is looking in field 100$a which does not exist. You see in logs the error : Unsupported UNIMARC character encoding [] for XML output for UNIMARC; 100$a This patch adds the format "USMARC" to XML serialization, like in C4::Items::_get_unlinked_subfields_xml Test plan : - On a UNIMARC database - Define a maching rule on title 200$a - Select a record with items - Export it using : Save as > MARC (Unicode/UTF-8) - Delete all items - Go to Tools > Stage MARC records for import - Upload exported file - Select title matching rule - Select "Ingore incoming record" in "Action if matching record found :" - Select Yes and "Always add items" in "Check for embedded item record data?" - Click Stage for import => Without patch you get the error => With patch the import is staged - Import into the catalog and check item is well recreated Signed-off-by: Amandine Zocca <azocca@ville-montauban.fr> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18757 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |martin.renvoize@ptfs-europe | |.com Status|Signed Off |Passed QA --- Comment #13 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Works as expected, passes qa scripts and code looks sound.. Passing QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18757 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |19.11.00 released in| | Status|Passed QA |Pushed to master -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18757 --- Comment #14 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Nice work! Pushed to master for 19.11.00 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18757 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |viliam.polonec@snk.sk --- Comment #15 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- *** Bug 23684 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org