[Bug 41681] New: bulkmarcimport.pl reports an incorrect number of MARC records processed
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41681 Bug ID: 41681 Summary: bulkmarcimport.pl reports an incorrect number of MARC records processed Initiative type: --- Sponsorship --- status: Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Architecture, internals, and plumbing Assignee: koha-bugs@lists.koha-community.org Reporter: hammat.wele@inlibro.com QA Contact: testopia@bugs.koha-community.org Created attachment 191849 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=191849&action=edit A marc file containing 5 records When running the script bulkmarcimport.pl, the number of MARC records reported at the end of the execution does not match the actual number of records processed from the input file. To reproduce: 1. Download the joined MARC file containing 5 records. 2. Run the import command: ./misc/migration_tools/bulkmarcimport.pl -b --file=marc_file.mrc 3. Check the final script report ==> 11 MARC records done in 0.566349983215332 seconds The reported number is incorrect -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41681 Hammat wele <hammat.wele@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |hammat.wele@inlibro.com |ity.org | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41681 --- Comment #1 from Hammat wele <hammat.wele@inlibro.com> --- Created attachment 191850 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=191850&action=edit Bug 41681: bulkmarcimport.pl reports an incorrect number of MARC records processed When running the script bulkmarcimport.pl, the number of MARC records reported at the end of the execution does not match the actual number of records processed from the input file. To test: 1. Download the joined MARC file containing 5 records. 2. Run the import command: ./misc/migration_tools/bulkmarcimport.pl -b --file=marc_file.mrc 3. Check the final script report ==> 11 MARC records done in 0.566349983215332 seconds Expected output should show 5 records 4. Apply the patch 5. Repeat step 2, 3 ==> The correct number is shown -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41681 Hammat wele <hammat.wele@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41681 Hammat wele <hammat.wele@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |caroline.cyr-la-rose@inlibr | |o.com, | |philippe.blouin@inlibro.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41681 Roman Dolny <roman.dolny@jezuici.pl> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=41147 CC| |roman.dolny@jezuici.pl -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41681 Janusz Kaczmarek <januszop@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41681 Janusz Kaczmarek <januszop@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #191850|0 |1 is obsolete| | --- Comment #2 from Janusz Kaczmarek <januszop@gmail.com> --- Created attachment 193360 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=193360&action=edit Bug 41681: bulkmarcimport.pl reports an incorrect number of MARC records processed When running the script bulkmarcimport.pl, the number of MARC records reported at the end of the execution does not match the actual number of records processed from the input file. To test: 1. Download the joined MARC file containing 5 records. 2. Run the import command: ./misc/migration_tools/bulkmarcimport.pl -b --file=marc_file.mrc 3. Check the final script report ==> 11 MARC records done in 0.566349983215332 seconds Expected output should show 5 records 4. Apply the patch 5. Repeat step 2, 3 ==> The correct number is shown Signed-off-by: Janusz Kaczmarek <januszop@gmail.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41681 Jan Kissig <bibliothek@th-wildau.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nick@bywatersolutions.com --- Comment #3 from Jan Kissig <bibliothek@th-wildau.de> --- *** Bug 41147 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41681 Jan Kissig <bibliothek@th-wildau.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bibliothek@th-wildau.de --- Comment #4 from Jan Kissig <bibliothek@th-wildau.de> --- Created attachment 194964 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=194964&action=edit File with 2 MARC records where the first is not valid -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41681 Jan Kissig <bibliothek@th-wildau.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Failed QA --- Comment #5 from Jan Kissig <bibliothek@th-wildau.de> ---
From bug 41147 there is an important comment from Nick in his patch:
# We increment here, if a record is skipped because we can't load form the batch or can't be converted # it still counts against the limit if a number to process is passed And so I created a file with the first record to be broken to test against param n=1 which would mean that only 1 record should be imported from the given file. As the first record is broken the script then should say 0 records imported (though 1 record processed) But the output is still like '1 MARC records done in ...' so I think there needs to be done a bit more or am I misinterpreting the number param to count only successful records? Tested via: perl misc/migration_tools/bulkmarcimport.pl -b -v -n=1 -m MARCXML --file broken.xml -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41681 Jan Kissig <bibliothek@th-wildau.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff Patch complexity|--- |Trivial patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41681 Jan Kissig <bibliothek@th-wildau.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #193360|0 |1 is obsolete| | --- Comment #6 from Jan Kissig <bibliothek@th-wildau.de> --- Created attachment 195081 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=195081&action=edit Bug 41681: bulkmarcimport.pl reports an incorrect number of MARC records processed When running the script bulkmarcimport.pl, the number of MARC records reported at the end of the execution does not match the actual number of records processed from the input file. To test: 1. Download the joined MARC file containing 5 records. 2. Run the import command: ./misc/migration_tools/bulkmarcimport.pl -b --file=marc_file.mrc 3. Check the final script report ==> 11 MARC records done in 0.566349983215332 seconds Expected output should show 5 records 4. Apply the patch 5. Repeat step 2, 3 ==> The correct number is shown Signed-off-by: Janusz Kaczmarek <januszop@gmail.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41681 --- Comment #7 from Jan Kissig <bibliothek@th-wildau.de> --- Created attachment 195082 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=195082&action=edit Bug 41681: (follow-up) Fix record number count when broken records are processed When running the script bulkmarcimport.pl, the number of MARC records reported at the end of the execution does not match the actual number of records processed from the input file. This patch fixes 3 issues - broken (invalid) records from batch get now counted (record_number) - when using the number param (n) to limit the number of records to be processed valid and invalid records get counted - when using th offset param (o) the scipt errored when broken records were part of the offset. To test: 1. Download the MARC file with the broken record 2. Run the following import commands and compare the counted import numbers: a) all records perl misc/migration_tools/bulkmarcimport.pl -b -v -m MARCXML --file 2_records_1st_broken.xml 4 MARC records done in 0.0421221256256104 seconds b) limit to 1 record in total perl misc/migration_tools/bulkmarcimport.pl -b -v -n 1 -m MARCXML --file 2_records_1st_broken.xml 4 MARC records done in 0.0420489311218262 seconds c) limit to 1 record offset 1 record perl misc/migration_tools/bulkmarcimport.pl -b -v -n 1 -o 1 -m MARCXML --file 2_records_1st_broken.xml :2: parser error : Opening and ending tag mismatch: leader2 line 2 and leader <leader2>00216nam a22001097a 4500</leader> 3. Apply the patch 4. Repeat steps in 2 and compare the results a) 2 MARC records done in 0.0405838489532471 seconds b) 1 MARC records done in 0.00948596000671387 seconds * c) 1 MARC records done in 0.0294840335845947 seconds *though this one was skipped -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41681 Jan Kissig <bibliothek@th-wildau.de> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=37564 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41681 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |nick@bywatersolutions.com |y.org | Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41681 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #195081|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41681 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #195082|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41681 --- Comment #8 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 199606 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199606&action=edit Bug 41681: bulkmarcimport.pl reports an incorrect number of MARC records processed When running the script bulkmarcimport.pl, the number of MARC records reported at the end of the execution does not match the actual number of records processed from the input file. To test: 1. Download the joined MARC file containing 5 records. 2. Run the import command: ./misc/migration_tools/bulkmarcimport.pl -b --file=marc_file.mrc 3. Check the final script report ==> 11 MARC records done in 0.566349983215332 seconds Expected output should show 5 records 4. Apply the patch 5. Repeat step 2, 3 ==> The correct number is shown Signed-off-by: Janusz Kaczmarek <januszop@gmail.com> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41681 --- Comment #9 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 199607 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199607&action=edit Bug 41681: (follow-up) Fix record number count when broken records are processed When running the script bulkmarcimport.pl, the number of MARC records reported at the end of the execution does not match the actual number of records processed from the input file. This patch fixes 3 issues - broken (invalid) records from batch get now counted (record_number) - when using the number param (n) to limit the number of records to be processed valid and invalid records get counted - when using th offset param (o) the scipt errored when broken records were part of the offset. To test: 1. Download the MARC file with the broken record 2. Run the following import commands and compare the counted import numbers: a) all records perl misc/migration_tools/bulkmarcimport.pl -b -v -m MARCXML --file 2_records_1st_broken.xml 1 MARC records done in 0.0421221256256104 seconds b) limit to 1 record in total perl misc/migration_tools/bulkmarcimport.pl -b -v -n 1 -m MARCXML --file 2_records_1st_broken.xml 1 MARC records done in 0.0420489311218262 seconds c) limit to 1 record offset 1 record perl misc/migration_tools/bulkmarcimport.pl -b -v -n 1 -o 1 -m MARCXML --file 2_records_1st_broken.xml :2: parser error : Opening and ending tag mismatch: leader2 line 2 and leader <leader2>00216nam a22001097a 4500</leader> 3. Apply the patch 4. Repeat steps in 2 and compare the results a) 2 MARC records done in 0.0405838489532471 seconds b) 1 MARC records done in 0.00948596000671387 seconds * c) 1 MARC records done in 0.0294840335845947 seconds *though this one was skipped Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41681 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA --- Comment #10 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Simple fixes, moving to PQA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41681 Jan Kissig <bibliothek@th-wildau.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@gmail.com --- Comment #11 from Jan Kissig <bibliothek@th-wildau.de> --- *** Bug 42782 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41681 Jonathan Druart <jonathan.druart@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |37020 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 [Bug 37020] bulkmarcimport gets killed when inserting large files -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41681 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |26.11.00 released in| | Status|Passed QA |Pushed to main -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41681 --- Comment #12 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Thanks everyone! Pushed to main for 26.11! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41681 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |rel_25_11_candidate CC| |dcook@prosentient.com.au --- Comment #13 from David Cook <dcook@prosentient.com.au> --- Was assisting someone with an import in 25.11 and noticed that $record_number gets incremented twice in misc/migration_tools/bulkmarcimport.pl, so this owuld be a good one to backport (I've added the keyword). -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41681 Lucas Gass (lukeg) <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|enhancement |normal CC| |lucas@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41681 Lucas Gass (lukeg) <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|26.11.00 |26.11.00,26.05.04 released in| | Status|Pushed to main |Pushed to stable -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41681 --- Comment #14 from Lucas Gass (lukeg) <lucas@bywatersolutions.com> --- Nice work everyone! Pushed to 26.05.x for the upcoming 26.05.03 release. -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org