https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29654 Bug ID: 29654 Summary: Add Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Tools Assignee: koha-bugs@lists.koha-community.org Reporter: glasklas@gmail.com QA Contact: testopia@bugs.koha-community.org -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29654 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Add |Add option to | |bulkmarimport.pl for | |matching on original id -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29654 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |29440 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 [Bug 29440] Refactor/clean up bulkmarcimport.pl -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29654 --- Comment #1 from David Gustafsson <glasklas@gmail.com> --- If I'm not mistaken, currently in bulkmarkimport.pl, if no match is found for authorities the incoming record is used for matching against an existing Koha authority, this is not done for biblio records though. Instead of this inconsistent behaviour it would be better if there was an option to explicitly set if matching on incoming record ids should be performed or not. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29654 --- Comment #2 from David Gustafsson <glasklas@gmail.com> --- Created attachment 128335 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=128335&action=edit Bug 29654 - Add match_record_id option to bulkmarimport.pl -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29654 --- Comment #3 from David Gustafsson <glasklas@gmail.com> --- Not yet ready for review, will later add how to test. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29654 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #128335|0 |1 is obsolete| | --- Comment #4 from David Gustafsson <glasklas@gmail.com> --- Created attachment 128365 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=128365&action=edit Bug 29654 - Add match_record_id option to bulkmarimport.pl Add -match_record_id option to bulkmarkimport.pl. When enabled incoming record id will be assumed to match record in Koha. To test: 1) Before applying patch, export a biblio from staff interace 2) Import using the exported file: ./bulkmarcimport.pl -m="MARCXML" -v -b -file bib-xxxxx.marcxml -insert -update -c=MARC21 -l "/tmp/import.log" 3) The import with be inserted and create a duplicate record in Koha. Check the import.log or search using the staff interface. (could also crash with duplicate key error if contains item) 4) Apply patch 5) Run the import again with the same command as above 6) A new record should been inserted 7) Remove all newly inserted duplicate records (it's important to keep the record the export was initially performed on) 6) Run the import again, but add the -match_record_id option 8) The original Koha record should now have been updated -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29654 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29654 --- Comment #5 from David Gustafsson <glasklas@gmail.com> --- Created attachment 128575 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=128575&action=edit Bug 29654: Add match_record_id option to bulkmarimport.pl Add -match_record_id option to bulkmarkimport.pl. When enabled incoming record id will be assumed to match record in Koha. To test: 1) Before applying patch, export a biblio from staff interace 2) Import using the exported file: ./bulkmarcimport.pl -m="MARCXML" -v -b -file bib-xxxxx.marcxml -insert -update -c=MARC21 -l "/tmp/import.log" 3) The import with be inserted and create a duplicate record in Koha. Check the import.log or search using the staff interface. (could also crash with duplicate key error if contains item) 4) Apply patch 5) Run the import again with the same command as above 6) A new record should been inserted 7) Remove all newly inserted duplicate records (it's important to keep the record the export was initially performed on) 6) Run the import again, but add the -match_record_id option 8) The original Koha record should now have been updated -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29654 Frank Hansen <frank.hansen@ub.lu.se> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |frank.hansen@ub.lu.se -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29654 Frank Hansen <frank.hansen@ub.lu.se> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Patch doesn't apply --- Comment #6 from Frank Hansen <frank.hansen@ub.lu.se> --- The patch doesn't apply anymore. Rebase please. Using index info to reconstruct a base tree... M misc/migration_tools/bulkmarcimport.pl Falling back to patching base and 3-way merge... Auto-merging misc/migration_tools/bulkmarcimport.pl CONFLICT (content): Merge conflict in misc/migration_tools/bulkmarcimport.pl error: Failed to merge in the changes. Patch failed at 0001 Bug 29654: Add match_record_id option to bulkmarimport.pl hint: Use 'git am --show-current-patch=diff' to see the failed patch When you have resolved this problem run "git bz apply --continue". If you would prefer to skip this patch, instead run "git bz apply --skip". To restore the original branch and stop patching run "git bz apply --abort". Patch left in /tmp/Bug-29654-Add-matchrecordid-option-to-bulkmarimpor-rJcwEW.patch vagrant@kohadevbox:kohaclone(bug29654|AM 1/1)$ -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29654 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #128365|0 |1 is obsolete| | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29654 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Needs Signoff --- Comment #7 from David Gustafsson <glasklas@gmail.com> --- I must have accidentally added two different iterations of the same patch, now it should work. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29654 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Patch doesn't apply --- Comment #8 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Sorry, patch doesn't apply (again): Apply? [(y)es, (n)o, (i)nteractive] y Applying: Bug 29440: Refactor and clean up bulkmarcimport.pl Using index info to reconstruct a base tree... M C4/Biblio.pm Falling back to patching base and 3-way merge... Auto-merging C4/Biblio.pm CONFLICT (content): Merge conflict in C4/Biblio.pm error: Failed to merge in the changes. Patch failed at 0001 Bug 29440: Refactor and clean up bulkmarcimport.pl hint: Use 'git am --show-current-patch=diff' to see the failed patch When you have resolved this problem run "git bz apply --continue". If you would prefer to skip this patch, instead run "git bz apply --skip". To restore the original branch and stop patching run "git bz apply --abort". Patch left in /tmp/Bug-29440-Refactor-and-clean-up-bulkmarcimportpl-zsN_GN.patch Cannot apply cleanly patches from bug 29440. Everything will be left dirty. git bz apply --continue will not continue the process if patches from other bug reports need to be applied. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29654 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Needs Signoff --- Comment #9 from David Gustafsson <glasklas@gmail.com> --- Sorry about being slow to reply. Tried applying the patch and applied cleanly for me, realized the conflict was in bug 29440 which since have been rebased. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29654 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |glasklas@gmail.com |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29654 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Patch doesn't apply -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29654 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |BLOCKED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29654 M <schodkowy.omegi-0r@icloud.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |schodkowy.omegi-0r@icloud.c | |om --- Comment #10 from M <schodkowy.omegi-0r@icloud.com> --- I'm not sure, as far as I understand the code, that this is the best approach to implementing it. It looks like it still tries to match the record by old logic, and only afterwards swaps out the ID. I don't think it should perform any matching when we specify the Koha record ID upfront, or at most it should check if it already exists... -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29654 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |martin.renvoize@ptfs-europe | |.com Status|BLOCKED |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29654 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Patch doesn't apply CC| |david@davidnind.com --- Comment #11 from David Nind <david@davidnind.com> --- The patch no longer applies - sha1/fake ancestor error: git bz apply 29654 Bug 29654 - Add option to bulkmarimport.pl for matching on original id 128575 - Bug 29654: Add match_record_id option to bulkmarimport.pl Apply? [(y)es, (n)o, (i)nteractive] y Applying: Bug 29654: Add match_record_id option to bulkmarimport.pl error: sha1 information is lacking or useless (misc/migration_tools/bulkmarcimport.pl). error: could not build fake ancestor Patch failed at 0001 Bug 29654: Add match_record_id option to bulkmarimport.pl -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29654 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #128575|0 |1 is obsolete| | --- Comment #12 from David Gustafsson <glasklas@gmail.com> --- Created attachment 178169 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=178169&action=edit Bug 29654: Add match_record_id option to bulkmarimport.pl Add -match_record_id option to bulkmarkimport.pl. When enabled incoming record id will be assumed to match record in Koha. To test: 1) Before applying patch, export a biblio from staff interace 2) Import using the exported file: ./bulkmarcimport.pl -m="MARCXML" -v -b -file bib-xxxxx.marcxml -insert -update -c=MARC21 -l "/tmp/import.log" 3) The import with be inserted and create a duplicate record in Koha. Check the import.log or search using the staff interface. (could also crash with duplicate key error if contains item) 4) Apply patch 5) Run the import again with the same command as above 6) A new record should been inserted 7) Remove all newly inserted duplicate records (it's important to keep the record the export was initially performed on) 6) Run the import again, but add the -match_record_id option 8) The original Koha record should now have been updated Sponsored-by: Gothenburg University Library -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29654 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Needs Signoff --- Comment #13 from David Gustafsson <glasklas@gmail.com> --- Rebased against main. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29654 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29654 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #178169|0 |1 is obsolete| | --- Comment #14 from David Nind <david@davidnind.com> --- Created attachment 182239 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=182239&action=edit Bug 29654: Add match_record_id option to bulkmarimport.pl Add -match_record_id option to bulkmarkimport.pl. When enabled incoming record id will be assumed to match record in Koha. To test: 1) Before applying patch, export a biblio from staff interace 2) Import using the exported file: ./bulkmarcimport.pl -m="MARCXML" -v -b -file bib-xxxxx.marcxml -insert -update -c=MARC21 -l "/tmp/import.log" 3) The import with be inserted and create a duplicate record in Koha. Check the import.log or search using the staff interface. (could also crash with duplicate key error if contains item) 4) Apply patch 5) Run the import again with the same command as above 6) A new record should been inserted 7) Remove all newly inserted duplicate records (it's important to keep the record the export was initially performed on) 6) Run the import again, but add the -match_record_id option 8) The original Koha record should now have been updated Sponsored-by: Gothenburg University Library Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29654 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the| |This enhancement adds a release notes| |`match_record_id` option to | |the | |./misc/migration_tools/bulk | |marcimport.pl script. If a | |record matches based on the | |ID, it is updated. | |Previously, without this | |option, a record (but not | |the items) would be | |duplicated. | | | |Example use | |case: allows updates to | |previously exported | |records, without creating | |duplicate records. --- Comment #15 from David Nind <david@davidnind.com> --- Testing notes (using KTD): 1. Running the command in step 2 generates this output - a duplicate record is inserted, but no item (I used record 262 - Programming perl): ./misc/migration_tools/bulkmarcimport.pl -m="MARCXML" -v -b -file bib-262.marcxml -insert -update -c=MARC21 -l "/tmp/import.log" Using --update without --match or --isbn seems to be useless. Characteristic MARC flavour: MARC21 Use of uninitialized value in concatenation (.) or string at /usr/share/perl5/MARC/File/XML.pm line 399, <GEN3> chunk 2. .DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::mysql::st execute failed: Duplicate entry '578' for key 'PRIMARY' at /kohadevbox/koha/Koha/Object.pm line 174 ERROR: Adding items to bib 439 failed: Duplicate ID at ./misc/migration_tools/bulkmarcimport.pl line 636. 1 MARC records done in 0.155983924865723 seconds /tmp/import.log: 1 id;operation;status 2 262;insert;ok 3 439;insert items;ERROR 4 file : bib-262.marcxml 5 1 MARC records done in 0.155983924865723 seconds 2. The patch still applies. 3. Running the command again for step 5 generates this output - another record is inserted without an item (so have two duplicate records): ./misc/migration_tools/bulkmarcimport.pl -m="MARCXML" -v -b -file bib-262.marcxml -insert -update -c=MARC21 -l "/tmp/import.log" Using --update without --match or --isbn seems to be useless. Characteristic MARC flavour: MARC21 Use of uninitialized value in concatenation (.) or string at /usr/share/perl5/MARC/File/XML.pm line 399, <GEN3> chunk 2. .DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::mysql::st execute failed: Duplicate entry '578' for key 'PRIMARY' at /kohadevbox/koha/Koha/Object.pm line 174 ERROR: Adding items to bib 440 failed: Duplicate ID at ./misc/migration_tools/bulkmarcimport.pl line 642. 1 MARC records done in 0.161586046218872 seconds /tmp/import.log: 1 id;operation;status 2 262;insert;ok 3 440;insert items;ERROR 4 file : bib-262.marcxml 5 1 MARC records done in 0.161586046218872 seconds 4. Running the command again with the -match_record_id option (after deleting the duplicate records) generates this output: ./misc/migration_tools/bulkmarcimport.pl -m="MARCXML" -v -b -file bib-262.marcxml -insert -update -c=MARC21 -match_record_id -l "/tmp/import.log" Using --update without --match or --isbn seems to be useless. Characteristic MARC flavour: MARC21 Use of uninitialized value in concatenation (.) or string at /usr/share/perl5/MARC/File/XML.pm line 399, <GEN3> chunk 2. . 1 MARC records done in 0.153377056121826 seconds /tmp/import.log: 1 id;operation;status 2 262;update;ok 3 file : bib-262.marcxml 4 1 MARC records done in 0.153377056121826 seconds 5. Note that with CataloguingLog system preference enabled (it is on by default in KTD), there is no record of any changes to the record - "No log found for Bibliographic record 262.". 6. I made a change to the exported record (edited the description in 504$a), and this change was reflected in the record. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29654 Bug 29654 depends on bug 29440, which changed state. Bug 29440 Summary: Refactor/clean up bulkmarcimport.pl https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to main |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29654 Lucas Gass (lukeg) <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Patch doesn't apply CC| |lucas@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29654 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #182239|0 |1 is obsolete| | --- Comment #16 from David Gustafsson <glasklas@gmail.com> --- Created attachment 189827 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=189827&action=edit Bug 29654: Add match_record_id option to bulkmarimport.pl Add -match_record_id option to bulkmarkimport.pl. When enabled incoming record id will be assumed to match record in Koha. To test: 1) Before applying patch, export a biblio from staff interace 2) Import using the exported file: ./bulkmarcimport.pl -m="MARCXML" -v -b -file bib-xxxxx.marcxml -insert -update -c=MARC21 -l "/tmp/import.log" 3) The import with be inserted and create a duplicate record in Koha. Check the import.log or search using the staff interface. (could also crash with duplicate key error if contains item) 4) Apply patch 5) Run the import again with the same command as above 6) A new record should been inserted 7) Remove all newly inserted duplicate records (it's important to keep the record the export was initially performed on) 6) Run the import again, but add the -match_record_id option 8) The original Koha record should now have been updated Sponsored-by: Gothenburg University Library Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29654 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29654 --- Comment #17 from David Gustafsson <glasklas@gmail.com> --- Rebased against master. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29654 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29654 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #189827|0 |1 is obsolete| | --- Comment #18 from David Nind <david@davidnind.com> --- Created attachment 194211 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=194211&action=edit Bug 29654: Add match_record_id option to bulkmarimport.pl Add -match_record_id option to bulkmarkimport.pl. When enabled incoming record id will be assumed to match record in Koha. To test: 1) Before applying patch, export a biblio from staff interace 2) Import using the exported file: ./bulkmarcimport.pl -m="MARCXML" -v -b -file bib-xxxxx.marcxml -insert -update -c=MARC21 -l "/tmp/import.log" 3) The import with be inserted and create a duplicate record in Koha. Check the import.log or search using the staff interface. (could also crash with duplicate key error if contains item) 4) Apply patch 5) Run the import again with the same command as above 6) A new record should been inserted 7) Remove all newly inserted duplicate records (it's important to keep the record the export was initially performed on) 6) Run the import again, but add the -match_record_id option 8) The original Koha record should now have been updated Sponsored-by: Gothenburg University Library Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29654 --- Comment #19 from David Nind <david@davidnind.com> --- (In reply to David Gustafsson from comment #17)
Rebased against master.
Things still work as per the test plan. With the rebase, you could have changed the status back to signoff (rather than needing a new signoff) 8-) -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org