Merging authority records can corrupt data
Hi! It seems that merging authorities with lots of linked biblios can lead to data corruption (at least in 21.05): The merging of the auth record itself seems to work, but the linking of the biblios from old to new auth record happens inside a normal synchrone web request. If this requests times out (because there might be a lot of biblio records to relink), some of the biblios still point to the old auth, which is already deleted. (At least this seems to be happening, based on the complaints we got from librarians) I guess the only fix is to handle authority merging via background jobs / job queue. Is this a known bug? I couldn't find anything about that in bugzilla? If yes, is there a timeframe when it will be fixed? If no, I will create a bug (though reproducing it will need some weird fixtures in the DB..). In any case, the Steiermärkische Landesbibliothek is keen to get this working, so I could give fixing it a try. But this will need some discussion, I assume? As I suppose this will be a bit more work than a single-line fix in a template, I'd like to have some backing by other Koha-devs, that this behaviour is indeed a bug, and that using background jobs is the correct way to fix it. Greetings, domm -- #!/usr/bin/perl https://domm.plix.at for(ref bless{},just'another'perl'hacker){s-:+-$"-g&&print$_.$/}
Hi Thomas, Not sure if this will help but what is the value of the *AuthorityMergeLimit* sys pref? I think the default value is 50 and any authority being merged with more than 50 linked biblios will be processed by the *misc/migration_tools/merge_authority.pl <http://merge_authority.pl>* cronjob (which runs daily by default if I'm not mistaken). If the problem you're experiencing is indeed related to having lots of linked biblios, maybe lowering this value (or setting it if empty) might help? In the past I reported an issue with subsequent authority merging (that get queued to be processed by the cronjob): https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22437 But I believe this has been fixed. On Fri, 30 Sept 2022 at 12:19, Thomas Klausner <domm@plix.at> wrote:
Hi!
It seems that merging authorities with lots of linked biblios can lead to data corruption (at least in 21.05):
The merging of the auth record itself seems to work, but the linking of the biblios from old to new auth record happens inside a normal synchrone web request. If this requests times out (because there might be a lot of biblio records to relink), some of the biblios still point to the old auth, which is already deleted. (At least this seems to be happening, based on the complaints we got from librarians)
I guess the only fix is to handle authority merging via background jobs / job queue.
Is this a known bug? I couldn't find anything about that in bugzilla? If yes, is there a timeframe when it will be fixed? If no, I will create a bug (though reproducing it will need some weird fixtures in the DB..).
In any case, the Steiermärkische Landesbibliothek is keen to get this working, so I could give fixing it a try. But this will need some discussion, I assume? As I suppose this will be a bit more work than a single-line fix in a template, I'd like to have some backing by other Koha-devs, that this behaviour is indeed a bug, and that using background jobs is the correct way to fix it.
Greetings, domm
-- #!/usr/bin/perl https://domm.plix.at for(ref bless{},just'another'perl'hacker){s-:+-$"-g&&print$_.$/} _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
Hi! On Fri, Sep 30, 2022 at 02:11:13PM +0000, Pedro Amorim wrote:
Not sure if this will help but what is the value of the *AuthorityMergeLimit* sys pref? I think the default value is 50 and any authority being merged with more than 50 linked biblios will be processed by the *misc/migration_tools/merge_authority.pl <http://merge_authority.pl>* cronjob (which runs daily by default if I'm not mistaken).
I didn't know about that setting (or the cronjob), but it is set to 50. But we'll try to set it to a lower value (and make sure that the cronjob is set up).
In the past I reported an issue with subsequent authority merging (that get queued to be processed by the cronjob): https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22437
But I believe this has been fixed.
ok, thanks, I'll also take a look at this bug and it's fix. Thanks a lot! Greetings, domm -- #!/usr/bin/perl https://domm.plix.at for(ref bless{},just'another'perl'hacker){s-:+-$"-g&&print$_.$/}
participants (2)
-
Pedro Amorim -
Thomas Klausner