[Bug 35591] New: bulkmarcimport.pl overwrites data if controlnumber eq authid
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35591 Bug ID: 35591 Summary: bulkmarcimport.pl overwrites data if controlnumber eq authid Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: MARC Authority data support Assignee: koha-bugs@lists.koha-community.org Reporter: domm@plix.at QA Contact: testopia@bugs.koha-community.org We had a lot of auth_header (~700.000). Then we decided to import a lot more (10.000.000). We used --update together with a match-rule (based on 024) to prevent duplicates. Which all worked well. Until... We discovered that ~30k of our old auths where overwritten with "random" data (some other auths). What used to be "Graz" was now "Centro Editoriale Valtortiano". After a bit of panic we were able to restore our old auths from a backup. But what happend? The problem is that eg "Graz" had authid=777317. But "Valtortiano" has 001=00777317X. If you squint a bit (and ignore the leading 0s and the X at the end) that's the same number! And it seems that C4::AuthoritiesMarc::GuessAuthId($record) returns the value of 001, which somehow causes the confusion. I'm not entirely sure yet, where the X is stripped, but the effect is that auth-record 777317 is overwritten. An easy fix would be to add a new command line param to bulkmarcimport to either disable the call to GuessAuthId (which keeps with backwards compat, but makes it easy for other people to also run into this problem); or to only call GuessAuthId if the new param is set (saver, but changes behaviour) If you agree that this is a problem, I can prepare a patch. If we did something stupid, please enlighten us! -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35591 --- Comment #1 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Overall this doesn't sound like intended behavior. Koha did not always replace 001 with the authid, in the distant past it kept the imported 001. I could imagine the behavior made more sense back then, but does much less now. I see 2 different bugs here: * It breaks importing for everyone that imports data with an existing 001 that is somewhat numerical and can clash with the authids in Koha that are added to 001 on import. * The guess is not even exact. I assume this is because of using the search index, but maybe there should be an additional Perl check if they are really equivalent before overlay? I would not be opposed to change the behavior here or make it a legacy switch that needs to be chosen actively. The current behavior appears quite dangerous. Some other thoughts: Could it make sense to forgo GuessAuthId when another match-rule is used? And maybe have a separate bug for making the match more exact/smarter. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35591 --- Comment #2 from Thomas Klausner <domm@plix.at> --- Created attachment 161303 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=161303&action=edit handcrafted auth marcxml with controlnumber 1402 -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35591 --- Comment #3 from Thomas Klausner <domm@plix.at> --- Created attachment 161304 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=161304&action=edit auth_1402_pre_import.png - screenshot before import -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35591 --- Comment #4 from Thomas Klausner <domm@plix.at> --- Created attachment 161305 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=161305&action=edit auth_1402_post_import.png - screenshot after import -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35591 --- Comment #5 from Thomas Klausner <domm@plix.at> --- Here's some more info and tests for this bug: * twin_040682714.xml is this GND record http://d-nb.info/gnd/4068271-7 ("Zwilling", i.e. twin), but modified to have <controlfield tag="001">1402</controlfield>, which matches the authid 1402 "Aristotle" in koha-testing-docker * auth_1402_pre_import.png is a screenshot of authid 1402 from koha-testing-docker * copy twin_040682714.xml to the root of your koha-testing-docker (so it will be available inside the container) * run: misc/migration_tools/bulkmarcimport.pl -m MARCXML -c utf8 -a --file twin_040682714.xml --all --match isbn,001 (you can also run with --match, makes no difference, as long as --match does not actually hit a match) * auth_1402_post_import.png is a screenshot of authid 1402 after running bulkmarcimport, showing that the authid 1402 now contains the data from twin_040682714.xml -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35591 --- Comment #6 from Thomas Klausner <domm@plix.at> --- Created attachment 161306 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=161306&action=edit Bug 35591: Prevent bulkmarcimport from replacing local auth_header If you run bulkmarcimport,pl on authorities with --update (or --all) it will overwrite a local auth_header if the auth_id of a local auth_header matches the control-number (field 001) of an imported MARC record. This patch adds a new option --guessauthid (which is still enabled by default to keep the script backwards compatible, but broken), which can turn off this behaviour if called as --noguessauthid. To test with koha-testing-docker: * Inspect cgi-bin/koha/authorities/detail.pl?authid=1402 which should be PERSO_NAME of Aristotles. * get the file twin_040682714.xml from Bugzilla and make it available inside of koha-testing-docker * run: misc/migration_tools/bulkmarcimport.pl -m MARCXML -c utf8 -a --file twin_040682714.xml --all --match isbn,001 * Check cgi-bin/koha/authorities/detail.pl?authid=1402, which now is a TOPIC_TERM "Zwilling" (twin...) Reset koha-testing-docker, apply the patch * run with the new flag misc/migration_tools/bulkmarcimport.pl -m MARCXML -c utf8 -a --file twin_040682714.xml --all --match isbn,001 --noguessauthid * cgi-bin/koha/authorities/detail.pl?authid=1402 still is "Aristotle" * "Zwilling" is cgi-bin/koha/authorities/detail.pl?authid=1708 (at least on my ktd, the actual id might vary) -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35591 Thomas Klausner <domm@plix.at> changed: What |Removed |Added ---------------------------------------------------------------------------- Change sponsored?|--- |Sponsored --- Comment #7 from Thomas Klausner <domm@plix.at> --- Sponsored-by: Steiermärkische Landesbibliothek -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35591 Thomas Klausner <domm@plix.at> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|--- |Small patch -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35591 Thomas Klausner <domm@plix.at> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |domm@plix.at |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35591 Thomas Klausner <domm@plix.at> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35591 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #161306|0 |1 is obsolete| | --- Comment #8 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Created attachment 162409 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=162409&action=edit Bug 35591: Prevent bulkmarcimport from replacing local auth_header If you run bulkmarcimport,pl on authorities with --update (or --all) it will overwrite a local auth_header if the auth_id of a local auth_header matches the control-number (field 001) of an imported MARC record. This patch adds a new option --guessauthid (which is still enabled by default to keep the script backwards compatible, but broken), which can turn off this behaviour if called as --noguessauthid. To test with koha-testing-docker: * Inspect cgi-bin/koha/authorities/detail.pl?authid=1402 which should be PERSO_NAME of Aristotles. * get the file twin_040682714.xml from Bugzilla and make it available inside of koha-testing-docker * run: misc/migration_tools/bulkmarcimport.pl -m MARCXML -c utf8 -a --file twin_040682714.xml --all --match isbn,001 * Check cgi-bin/koha/authorities/detail.pl?authid=1402, which now is a TOPIC_TERM "Zwilling" (twin...) Reset koha-testing-docker, apply the patch * run with the new flag misc/migration_tools/bulkmarcimport.pl -m MARCXML -c utf8 -a --file twin_040682714.xml --all --match isbn,001 --noguessauthid * cgi-bin/koha/authorities/detail.pl?authid=1402 still is "Aristotle" * "Zwilling" is cgi-bin/koha/authorities/detail.pl?authid=1708 (at least on my ktd, the actual id might vary) Sponsored-by: Steiermärkische Landesbibliothek Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35591 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off CC| |victor@tuxayo.net --- Comment #9 from Victor Grousset/tuxayo <victor@tuxayo.net> --- It works :) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35591 --- Comment #10 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Well done, it's a nasty bug. (In reply to Katrin Fischer from comment #1)
I would not be opposed to change the behavior here or make it a legacy switch that needs to be chosen actively. The current behavior appears quite dangerous.
+1 for legacy switch. The current patch is a fix switch. Default is still dangerous. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35591 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Patch doesn't apply CC| |nick@bywatersolutions.com --- Comment #11 from Nick Clemens <nick@bywatersolutions.com> --- I believe bug 29440 fixed or altered this - the patch doesn't apply, nor do I recreate the behaviour following the test plan - GuessAuthId is no longer used in the script -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35591 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |In Discussion --- Comment #12 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Confirmed! I can reproduce on the commit before bug 29440. And I can't with the commits of bug 29440. Is there a worry to have about the broken legacy behavior that is "lost". -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org