[Bug 13167] New: Stage MARC for Import hangs
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13167 Bug ID: 13167 Summary: Stage MARC for Import hangs Change sponsored?: --- Product: Koha Version: 3.16 Hardware: All OS: All Status: NEW Severity: minor Priority: P5 - low Component: Tools Assignee: gmcharlt@gmail.com Reporter: tiburce.euphrasie@aupaysdescrevettes.org QA Contact: testopia@bugs.koha-community.org Created attachment 32969 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=32969&action=edit Zip contains 2 UNIMARC ISO-8859-1 records + a scan of the ISBN on the original book Aloha, I have detected un problem while trying to import a large set of UNIMARC records. I tried with a Koha 3.16.01 and a Koha 3.16.03, same issue : If the ISBN of a UNIMARC record begins with 979 then the 'Stage MARC for import' hangs. If I use the same UNIMARC record and change 979 to 978 in the ISBN, 'Stage MARC for import' works perfectly. You can find 2 UNIMARC Records (ISO8859-1) in the attachment if you wish to make tests by yourself (same records, one with ISBN 979-..., one with ISBN 978-...). I think I've also detected a small problem in cataloguing::value_builder::unimarc_field_010 : [...] Line 94 to 96 : if (length ($isbn) == 13){ $isbn = substr($isbn, 3, length($isbn)-3); } [...] Then, line 143 : $sth->execute($seg2, "978$seg2"); This "978" on line 143 is hard-coded and there's no relation with the original data inside $isbn. I don't know if this is a real problem but it might be better to keep the original data than replacing it with "978". Aloha. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13167 Frédéric Demians <frederic@tamil.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |frederic@tamil.fr Summary|Stage MARC for Import hangs |Stage MARC for Import hangs | |for biblio containing | |invalid ISBN-13 -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13167 Frédéric Demians <frederic@tamil.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff Patch complexity|--- |Trivial patch -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13167 --- Comment #1 from Frédéric Demians <frederic@tamil.fr> --- Created attachment 33130 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=33130&action=edit Bug 13167 Stage MARC for Import hangs for biblio containing invalid ISBN-13 If the ISBN of a UNIMARC record begins with 979 then the 'Stage MARC for import' hangs. If I use the same UNIMARC record and change 979 to 978 in the ISBN, 'Stage MARC for import' works perfectly. The patch deals with the fact that converting an ISBN-13 to ISBN-10 with Business::ISBN as_isbn10() method fails if the ISBN doesn't begin with 978. TEST PLAN: (1) Download, and decompress the ZIP file attached to BZ. (2) On a UNIMARC Koha instance, go in Tools > Stage MARC for import. (3) Choose the MARC file containing the record with an ISBN begining with 979. Click on Upload file, then Stage to import. (4) The Job progress bar stay at 0%. (5) Apply the patch. Repeat steps 2-3. The upload works. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13167 Frédéric Demians <frederic@tamil.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|gmcharlt@gmail.com |frederic@tamil.fr -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13167 --- Comment #2 from Frédéric Demians <frederic@tamil.fr> --- (In reply to Tiburce Euphrasie from comment #0)
I think I've also detected a small problem in
Open a separate bug report. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13167 Frédéric Demians <frederic@tamil.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Version|3.16 |master -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13167 Frédéric Demians <frederic@tamil.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ncubec@solusi.ac.zw --- Comment #3 from Frédéric Demians <frederic@tamil.fr> --- *** Bug 13158 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13167 --- Comment #4 from Tiburce Euphrasie <tiburce.euphrasie@aupaysdescrevettes.org> --- (In reply to Frédéric Demians from comment #2)
(In reply to Tiburce Euphrasie from comment #0)
I think I've also detected a small problem in
Open a separate bug report.
Aloha, Done. See bug 13192 : http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13192 Aloha. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13167 --- Comment #5 from Tiburce Euphrasie <tiburce.euphrasie@aupaysdescrevettes.org> --- (In reply to Frédéric Demians from comment #1)
Created attachment 33130 [details] [review] Bug 13167 Stage MARC for Import hangs for biblio containing invalid ISBN-13
If the ISBN of a UNIMARC record begins with 979 then the 'Stage MARC for import' hangs. If I use the same UNIMARC record and change 979 to 978 in the ISBN, 'Stage MARC for import' works perfectly.
The patch deals with the fact that converting an ISBN-13 to ISBN-10 with Business::ISBN as_isbn10() method fails if the ISBN doesn't begin with 978.
Aloha Frédéric Demians, Adding 'return unless $isbn;' on line 1683 in Koha.pm means that the sub NormalizeISBN will return nothing for the specific ISBN sample I've provided. This is not a proper way to fix this bug because that patch implies that all those '979.*' ISBN simply don't exist. Bu the way, the initial summary for this bug was "Stage MARC for Import hangs", a mistake was done when changing this summary to "Stage MARC for Import hangs for biblio containing invalid ISBN-13", so I'll change it to "Stage MARC for Import hangs for biblio containing valid 979* ISBN-13" after redacting this comment. This ISBN is valid : - there's a scan from the original book in the attachment file 32969 : http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=32969 - There's some documentation available this kind of ISBN : http://www.afnil.org/ http://fr.wikipedia.org/wiki/International_Standard_Book_Number Aloha. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13167 Tiburce Euphrasie <tiburce.euphrasie@aupaysdescrevettes.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Stage MARC for Import hangs |Stage MARC for Import hangs |for biblio containing |for biblio containing valid |invalid ISBN-13 |979* ISBN-13 -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13167 --- Comment #6 from Frédéric Demians <frederic@tamil.fr> --- (In reply to Tiburce Euphrasie from comment #5) Aloha!
Aloha Frédéric Demians, Adding 'return unless $isbn;' on line 1683 in Koha.pm means that the sub NormalizeISBN will return nothing for the specific ISBN sample I've provided.
This is not a proper way to fix this bug because that patch implies that all those '979.*' ISBN simply don't exist.
Your assumption is not correct. The function can't transform your ISBN-13 into ISBN-10, because it begins with '979'. Rather than returning undef to say that the conversion is not possible, the function was failing badly, stopping completely the Perl process. That's what is corrected by the patch. Have you tried the patch? Does it solve the bug reported here? Can you import your record? Is your ISBN field preserved? For both records? The Koha internal function NormalizeISBN is called from other part of the Koha code base (see bug 13158). When the action requested to the function could not be completed, the function is supposed to return undef (nothing). That's exactly what'is done with the patch. The import process call NormalizeISBN function for whatever reason. I can't say why, indeed. And for doing nothing with the result, since ISBN field is not modified during import. But this is another story, and can't be fixed here.
Bu the way, the initial summary for this bug was "Stage MARC for Import hangs", a mistake was done when changing this summary to "Stage MARC for Import hangs for biblio containing invalid ISBN-13", so I'll change it to "Stage MARC for Import hangs for biblio containing valid 979* ISBN-13" after redacting this comment.
You're correct. That's a valid ISBN-13 which can't be converted into ISBN-10. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13167 --- Comment #7 from Tiburce Euphrasie <tiburce.euphrasie@aupaysdescrevettes.org> --- (In reply to Frédéric Demians from comment #6)
(In reply to Tiburce Euphrasie from comment #5)
Aloha! [...] Your assumption is not correct. The function can't transform your ISBN-13 into ISBN-10, because it begins with '979'. Rather than returning undef to say that the conversion is not possible, the function was failing badly, stopping completely the Perl process. That's what is corrected by the patch.
Have you tried the patch? Does it solve the bug reported here? Can you import your record? Is your ISBN field preserved? For both records? [...]
Aloha, Yes, it works, I tried it on a Koha 3.16.03 and there's no problem : I can import both records, I can find them and the ISBN are preserved. Thank you again for patching this bug, I can't test it on the master branch for now but it works perfectly for me. Aloha. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13167 --- Comment #8 from Frédéric Demians <frederic@tamil.fr> --- (In reply to Tiburce Euphrasie from comment #7)
Aloha, Yes, it works, I tried it on a Koha 3.16.03 and there's no problem : I can import both records, I can find them and the ISBN are preserved.
Thanks for confirming it works for you. It will help to get this patch back to Koha main code repository. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13167 Colin Campbell <colin.campbell@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #33130|0 |1 is obsolete| | CC| |colin.campbell@ptfs-europe. | |com --- Comment #9 from Colin Campbell <colin.campbell@ptfs-europe.com> --- Created attachment 34414 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=34414&action=edit Signed Off patch This is not Unimarc specific I reproduced this using marc-21. its a simple logic error in the NormalizeISBN routine while all valid 10 character isbns can be encoded as 13 character isbns the reverse is not true. Any process using NormalizeISBN is liable to error if it encounters this case not just Import (although thats where I encountered it independently ) -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13167 Nick Clemens <nick@quecheelibrary.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off CC| |nick@quecheelibrary.org --- Comment #10 from Nick Clemens <nick@quecheelibrary.org> --- Marking as signed off, it looks like that was Colin's intention -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13167 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13167 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #34414|0 |1 is obsolete| | --- Comment #11 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Created attachment 34699 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=34699&action=edit [PASSED QA] Bug 13167 Stage MARC for Import hangs for biblio containing invalid ISBN-13 If the ISBN of a UNIMARC record begins with 979 then the 'Stage MARC for import' hangs. If I use the same UNIMARC record and change 979 to 978 in the ISBN, 'Stage MARC for import' works perfectly. The patch deals with the fact that converting an ISBN-13 to ISBN-10 with Business::ISBN as_isbn10() method fails if the ISBN doesn't begin with 978. TEST PLAN: (1) Download, and decompress the ZIP file attached to BZ. (2) On a UNIMARC Koha instance, go in Tools > Stage MARC for import. (3) Choose the MARC file containing the record with an ISBN begining with 979. Click on Upload file, then Stage to import. (4) The Job progress bar stay at 0%. (5) Apply the patch. Repeat steps 2-3. The upload works. Signed-off-by: Colin Campbell <colin.campbell@ptfs-europe.com> Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> Tested in a UNIMARC installation, confirmed that the patch fixes the problem. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13167 --- Comment #12 from Colin Campbell <colin.campbell@ptfs-europe.com> --- (In reply to Nick Clemens from comment #10)
Marking as signed off, it looks like that was Colin's intention
It was indeed. Just adding that this is not UNIMARC specific ouurs im marc21 too -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13167 --- Comment #13 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 34916 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=34916&action=edit Bug 13167: (QA followup) regression tests Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com> -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13167 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #34699|0 |1 is obsolete| | --- Comment #14 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 34917 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=34917&action=edit Bug 13167 Stage MARC for Import hangs for biblio containing invalid ISBN-13 If the ISBN of a UNIMARC record begins with 979 then the 'Stage MARC for import' hangs. If I use the same UNIMARC record and change 979 to 978 in the ISBN, 'Stage MARC for import' works perfectly. The patch deals with the fact that converting an ISBN-13 to ISBN-10 with Business::ISBN as_isbn10() method fails if the ISBN doesn't begin with 978. TEST PLAN: (1) Download, and decompress the ZIP file attached to BZ. (2) On a UNIMARC Koha instance, go in Tools > Stage MARC for import. (3) Choose the MARC file containing the record with an ISBN begining with 979. Click on Upload file, then Stage to import. (4) The Job progress bar stay at 0%. (5) Apply the patch. Repeat steps 2-3. The upload works. Signed-off-by: Colin Campbell <colin.campbell@ptfs-europe.com> Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> Tested in a UNIMARC installation, confirmed that the patch fixes the problem. Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com> -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13167 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to Master CC| |tomascohen@gmail.com --- Comment #15 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Patch pushed to master. Thanks Frederic -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13167 Chris Cormack <chris@bigballofwax.co.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to Master |Pushed to Stable CC| |chris@bigballofwax.co.nz --- Comment #16 from Chris Cormack <chris@bigballofwax.co.nz> --- Pushed to 3.18.x, will be in 3.18.3 -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13167 Mason James <mtj@kohaaloha.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mtj@kohaaloha.com --- Comment #17 from Mason James <mtj@kohaaloha.com> --- Pushed to 3.16.x, will be in 3.16.7 -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13167 Fridolin SOMERS <fridolyn.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to Stable |RESOLVED CC| |fridolyn.somers@biblibre.co | |m Resolution|--- |FIXED --- Comment #18 from Fridolin SOMERS <fridolyn.somers@biblibre.com> --- Not needed for 3.14.x. I set as resolved -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13167 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |kyle@bywatersolutions.com --- Comment #19 from Kyle M Hall <kyle@bywatersolutions.com> --- *** Bug 13641 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13167 Olli-Antti Kivilahti <olli-antti.kivilahti@jns.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |olli-antti.kivilahti@jns.fi --- Comment #20 from Olli-Antti Kivilahti <olli-antti.kivilahti@jns.fi> --- Hi there! I think this solution is not a good one. I think the ISBNs starting with 979 are also valid. This solution gets rid of valid ISBNs if they are of the new numberspace 979, see the wikipedia article: https://en.wikipedia.org/wiki/International_Standard_Book_Number#EAN_format_... It says that we are "moving to"/"using" 979 ISBNs as well. Certainly we are getting such ISBNs from our vendors. See Bug 14287 for a follow-up -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13167 Olli-Antti Kivilahti <olli-antti.kivilahti@jns.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |14287 -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13167 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |katrin.fischer@bsz-bw.de --- Comment #21 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Hi Olli, best is to file a new bug here, as this is marked RESOLVED FIXED and so already off the radar. Please link the new bug to this one. Thx! -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org