https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34739 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=29522 -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34739 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |m.de.rooy@rijksmuseum.nl --- Comment #1 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- There is a development about adding URIs from 024 in auth to bib record when merging. A Biblibre report, forgot the number. This would be an exception. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34739 Janusz Kaczmarek <januszop@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|--- |Small patch Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34739 --- Comment #4 from Janusz Kaczmarek <januszop@gmail.com> --- Created attachment 175666 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=175666&action=edit Bug 34739: Unit tests BTW, there is a tiny adjustment to one of the previous subtests to fit to the modified merge behaviour. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34739 --- Comment #5 from Janusz Kaczmarek <januszop@gmail.com> --- *** Bug 38742 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34739 Roman Dolny <roman.dolny@jezuici.pl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #175665|0 |1 is obsolete| | Attachment #175666|0 |1 is obsolete| | --- Comment #6 from Roman Dolny <roman.dolny@jezuici.pl> --- Created attachment 175667 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=175667&action=edit Bug 34739: Linked biblios should not be merged (updated) when changes to an authority don't change the authorized heading Currently, the function C4::AuthoritiesMarc::merge calls ModBiblio for all bibliographic records linked to the modified authority record. It could seem a just behavior but we know that this could lead to performance problems. This is why AuthorityMergeLimit systempreference with merge_authorities.pl was introduced (earlier: dontmerge). Although this partially solves the performance problems, for moderately large and large catalogs (500+ K) and authority records linked with a large number of bibliographic records, performance problems still occur, even with overnight processing with merge_authorities.pl. This problem is particularly apparent when updating authority records from central databases, whose modifications are beyond our control. It turns out, however, that many such modifications to authority records involve fields other than the heading field (1XX in MARC 21), so these updates in fact do not modify bibliographic records. However, with the current merge function, every touch on an authority record forces a call to the ModBiblio function. The number of relatively expensive ModBiblio calls could be reduced by checking whether there has been an actual change in the field associated with the authority record. However, with IncludeSeeFromInSearches or IncludeSeeAlsoFromInSearches preferences enabled, the record should be refreshed in the search engine just in case (should the 4XX, 5XX fields have changed, which we don't know at this point). Test plan: ========== 1. Make sure to have CataloguingLog set to Log and IncludeSeeFromInSearches set to Include. 2. Have a bibliographic record with an authority record linked. Open the authority record in a separate tab, open it in the editor and save, without modifying it. Repeat this 2-3 times. In the bibliographic record tab open the Modification log and see a number of entries - for each “empty” modification of the linked authority record. 3. Apply the patch ; restart_all. 4. Repat p. 2. Note that there are no new entries in the log. 5. Edit the authority record again, this time putting in one of the proper 4XX $a subfield an uncommon word (but do not touch the 1XX field). Save the authority record. In the bibliographic record tab control that there are no new entries in the modification log. Make a search for the word put in 4XX field. You should get the bibliographic record(s) linked to the edited authority record, even though the bibliographic record itself has not been modified. 6. Edit the authority record again, this time changing the 1XX field. Save the authority record. In the bibliographic record tab control that there are is new entry in the modification log. Sponsored-by: Ignatianum University in Cracow Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34739 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Failed QA --- Comment #8 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- # Subtest: ModBiblio calls from merge 1..4 not ok 1 - real modification of bibliographic record, ModBiblio called # Failed test 'real modification of bibliographic record, ModBiblio called' # at t/db_dependent/Authority/Merge.t line 612. # didn't find a warning # expected to find warning: ModBiblio called not ok 2 - no real modification of bibliographic record, index_records called # Failed test 'no real modification of bibliographic record, index_records called' # at t/db_dependent/Authority/Merge.t line 626. # didn't find a warning # expected to find warning: index_records called ok 3 - no real modification of bibliographic record, index_records not called not ok 4 - Only one bibliographic record updated while merging two authorities # Failed test 'Only one bibliographic record updated while merging two authorities' # at t/db_dependent/Authority/Merge.t line 655. # got: '0' # expected: '1' # Looks like you failed 3 tests of 4. not ok 13 - ModBiblio calls from merge -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34739 Janusz Kaczmarek <januszop@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Signed Off --- Comment #9 from Janusz Kaczmarek <januszop@gmail.com> --- (In reply to Marcel de Rooy from comment #8)

not ok 13 - ModBiblio calls from merge

This is somehow strange... Could you provide some more detailed information or repeat the test, please? I have tried anew on two separate fresh ktd installation -- everything looks clear: kohadev-koha@kohadevbox:koha(bug_34739)$ qa testing 2 commit(s) (applied to da08fc5 '1a Bug 31270: (follow-up) Fix descrip') Processing files before patches |========================>| 2 / 2 (100.00%) Processing files after patches |========================>| 2 / 2 (100.00%) OK C4/AuthoritiesMarc.pm OK t/db_dependent/Authority/Merge.t Processing additional checks OK! Running tests (1) * Proving /kohadevbox/koha/t/db_dependent/Authority/Merge.t OK! -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34739 --- Comment #11 from Janusz Kaczmarek <januszop@gmail.com> --- (In reply to Marcel de Rooy from comment #10)

Running tests (1) * Proving /usr/share/koha/t/db_dependent/Authority/Merge.t KO!

So it's strange. I don't know... I am helpless. Maybe you could have a look at the test -- is ModBiblio properly mocked in the test snippet added to Merge.t? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34739 --- Comment #13 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- And the following ones are new (iiuc): # Subtest: ModBiblio calls from merge 1..4 not ok 1 - real modification of bibliographic record, ModBiblio called # Failed test 'real modification of bibliographic record, ModBiblio called' # at t/db_dependent/Authority/Merge.t line 716. # didn't find a warning # expected to find warning: ModBiblio called not ok 2 - no real modification of bibliographic record, index_records called # Failed test 'no real modification of bibliographic record, index_records called' # at t/db_dependent/Authority/Merge.t line 730. # didn't find a warning # expected to find warning: index_records called ok 3 - no real modification of bibliographic record, index_records not called not ok 4 - Only one bibliographic record updated while merging two authorities # Failed test 'Only one bibliographic record updated while merging two authorities' # at t/db_dependent/Authority/Merge.t line 759. # got: '0' # expected: '1' # Looks like you failed 3 tests of 4. That could indeed be a bad mock or so ? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34739 --- Comment #15 from Janusz Kaczmarek <januszop@gmail.com> --- (In reply to Marcel de Rooy from comment #14)

+ my $biblio_module = Test::MockModule->new('C4::AuthoritiesMarc');

This is confusing btw :)

Maybe a little... This snippet is taken from one of the previous tests (cf. commit 3954808ac5052a53d5fa40502a4f7bd991be80ab). -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34739 Janusz Kaczmarek <januszop@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #175667|0 |1 is obsolete| | Attachment #175668|0 |1 is obsolete| | --- Comment #16 from Janusz Kaczmarek <januszop@gmail.com> --- Created attachment 179979 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=179979&action=edit Bug 34739: Linked biblios should not be merged (updated) when changes to an authority don't change the authorized heading Currently, the function C4::AuthoritiesMarc::merge calls ModBiblio for all bibliographic records linked to the modified authority record. It could seem a just behavior but we know that this could lead to performance problems. This is why AuthorityMergeLimit systempreference with merge_authorities.pl was introduced (earlier: dontmerge). Although this partially solves the performance problems, for moderately large and large catalogs (500+ K) and authority records linked with a large number of bibliographic records, performance problems still occur, even with overnight processing with merge_authorities.pl. This problem is particularly apparent when updating authority records from central databases, whose modifications are beyond our control. It turns out, however, that many such modifications to authority records involve fields other than the heading field (1XX in MARC 21), so these updates in fact do not modify bibliographic records. However, with the current merge function, every touch on an authority record forces a call to the ModBiblio function. The number of relatively expensive ModBiblio calls could be reduced by checking whether there has been an actual change in the field associated with the authority record. However, with IncludeSeeFromInSearches or IncludeSeeAlsoFromInSearches preferences enabled, the record should be refreshed in the search engine just in case (should the 4XX, 5XX fields have changed, which we don't know at this point). Test plan: ========== 1. Make sure to have CataloguingLog set to Log and IncludeSeeFromInSearches set to Include. 2. Have a bibliographic record with an authority record linked. Open the authority record in a separate tab, open it in the editor and save, without modifying it. Repeat this 2-3 times. In the bibliographic record tab open the Modification log and see a number of entries - for each “empty” modification of the linked authority record. 3. Apply the patch ; restart_all. 4. Repat p. 2. Note that there are no new entries in the log. 5. Edit the authority record again, this time putting in one of the proper 4XX $a subfield an uncommon word (but do not touch the 1XX field). Save the authority record. In the bibliographic record tab control that there are no new entries in the modification log. Make a search for the word put in 4XX field. You should get the bibliographic record(s) linked to the edited authority record, even though the bibliographic record itself has not been modified. 6. Edit the authority record again, this time changing the 1XX field. Save the authority record. In the bibliographic record tab control that there are is new entry in the modification log. Sponsored-by: Ignatianum University in Cracow Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34739 --- Comment #18 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to Marcel de Rooy from comment #14)

+ my $biblio_module = Test::MockModule->new('C4::AuthoritiesMarc');

This is confusing btw :)

Loading authorities and calling it biblio, I mean specifically :) -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34739 Janusz Kaczmarek <januszop@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #179979|0 |1 is obsolete| | --- Comment #20 from Janusz Kaczmarek <januszop@gmail.com> --- Created attachment 180083 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=180083&action=edit Bug 34739: Linked biblios should not be merged (updated) when changes to an authority don't change the authorized heading Currently, the function C4::AuthoritiesMarc::merge calls ModBiblio for all bibliographic records linked to the modified authority record. It could seem a just behavior but we know that this could lead to performance problems. This is why AuthorityMergeLimit systempreference with merge_authorities.pl was introduced (earlier: dontmerge). Although this partially solves the performance problems, for moderately large and large catalogs (500+ K) and authority records linked with a large number of bibliographic records, performance problems still occur, even with overnight processing with merge_authorities.pl. This problem is particularly apparent when updating authority records from central databases, whose modifications are beyond our control. It turns out, however, that many such modifications to authority records involve fields other than the heading field (1XX in MARC 21), so these updates in fact do not modify bibliographic records. However, with the current merge function, every touch on an authority record forces a call to the ModBiblio function. The number of relatively expensive ModBiblio calls could be reduced by checking whether there has been an actual change in the field associated with the authority record. However, with IncludeSeeFromInSearches or IncludeSeeAlsoFromInSearches preferences enabled, the record should be refreshed in the search engine just in case (should the 4XX, 5XX fields have changed, which we don't know at this point). Test plan: ========== 1. Make sure to have CataloguingLog set to Log and IncludeSeeFromInSearches set to Include. 2. Have a bibliographic record with an authority record linked. Open the authority record in a separate tab, open it in the editor and save, without modifying it. Repeat this 2-3 times. In the bibliographic record tab open the Modification log and see a number of entries - for each “empty” modification of the linked authority record. 3. Apply the patch ; restart_all. 4. Repat p. 2. Note that there are no new entries in the log. 5. Edit the authority record again, this time putting in one of the proper 4XX $a subfield an uncommon word (but do not touch the 1XX field). Save the authority record. In the bibliographic record tab control that there are no new entries in the modification log. Make a search for the word put in 4XX field. You should get the bibliographic record(s) linked to the edited authority record, even though the bibliographic record itself has not been modified. 6. Edit the authority record again, this time changing the 1XX field. Save the authority record. In the bibliographic record tab control that there are is new entry in the modification log. Sponsored-by: Ignatianum University in Cracow Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34739 Thomas Klausner <domm@plix.at> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34739 Thomas Klausner <domm@plix.at> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #180083|0 |1 is obsolete| | --- Comment #23 from Thomas Klausner <domm@plix.at> --- Created attachment 180188 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=180188&action=edit Bug 34739: Linked biblios should not be merged (updated) when changes to an authority don't change the authorized heading Currently, the function C4::AuthoritiesMarc::merge calls ModBiblio for all bibliographic records linked to the modified authority record. It could seem a just behavior but we know that this could lead to performance problems. This is why AuthorityMergeLimit systempreference with merge_authorities.pl was introduced (earlier: dontmerge). Although this partially solves the performance problems, for moderately large and large catalogs (500+ K) and authority records linked with a large number of bibliographic records, performance problems still occur, even with overnight processing with merge_authorities.pl. This problem is particularly apparent when updating authority records from central databases, whose modifications are beyond our control. It turns out, however, that many such modifications to authority records involve fields other than the heading field (1XX in MARC 21), so these updates in fact do not modify bibliographic records. However, with the current merge function, every touch on an authority record forces a call to the ModBiblio function. The number of relatively expensive ModBiblio calls could be reduced by checking whether there has been an actual change in the field associated with the authority record. However, with IncludeSeeFromInSearches or IncludeSeeAlsoFromInSearches preferences enabled, the record should be refreshed in the search engine just in case (should the 4XX, 5XX fields have changed, which we don't know at this point). Test plan: ========== 1. Make sure to have CataloguingLog set to Log and IncludeSeeFromInSearches set to Include. 2. Have a bibliographic record with an authority record linked. Open the authority record in a separate tab, open it in the editor and save, without modifying it. Repeat this 2-3 times. In the bibliographic record tab open the Modification log and see a number of entries - for each “empty” modification of the linked authority record. 3. Apply the patch ; restart_all. 4. Repat p. 2. Note that there are no new entries in the log. 5. Edit the authority record again, this time putting in one of the proper 4XX $a subfield an uncommon word (but do not touch the 1XX field). Save the authority record. In the bibliographic record tab control that there are no new entries in the modification log. Make a search for the word put in 4XX field. You should get the bibliographic record(s) linked to the edited authority record, even though the bibliographic record itself has not been modified. 6. Edit the authority record again, this time changing the 1XX field. Save the authority record. In the bibliographic record tab control that there are is new entry in the modification log. Sponsored-by: Ignatianum University in Cracow Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl> Signed-off-by: Thomas Klausner <domm@plix.at> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34739 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |25.05.00 released in| | Status|Passed QA |Pushed to main -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34739 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |additional_work_needed --- Comment #26 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- We have some broken tests, please check ASAP (lots waiting to be pushed): Subtest: ModBiblio calls from merge 1..5 not ok 1 - real modification of bibliographic record, ModBiblio called not ok 2 - no real modification of bibliographic record, index_records called ok 3 - no real modification of bibliographic record, index_records not called not ok 4 - real modification of bibliographic record, ModBiblio called not ok 5 - only one bibliographic record updated while merging two authorities -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34739 --- Comment #28 from Thomas Klausner <domm@plix.at> --- when I run prove -lv t/db_dependent/Biblio_holdsqueue.t or t/db_dependent/Biblio.t and then t/db_dependent/Authority/Merge.t I get the failures. If then do reset_all, Merge.t works again. But I haven't had time to take a closer look why this happens -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34739 --- Comment #30 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to Jonathan Druart from comment #29)

If you run t/db_dependent/Biblio.t first, then Merge.t will fail.

I guess there are modifications made by Biblio.t to the MSS (marc subfield structure), either in DB or cached.

I have noticed this suspicious stuff:

698 $biblio_record->insert_fields_ordered( 699 MARC::Field->new( '609', '1', '1', a => 'Brown,', 'c' => 'Father', 9 => $authid ) ); 700 my $biblionumber = AddBiblio( $biblio_record, '', { skip_record_index => 1 } );

if you dump $biblio_record->as_formatted you see the 609 field.

BUT, if you add the following line right after: my $test = Koha::Biblios->find($biblionumber)->metadata->record then dump $test->as_formatted

=> 609 is not there (??)

Sounds like a framework issue. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34739 Thomas Klausner <domm@plix.at> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |39700 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39700 [Bug 39700] Fix test case t/db_dependent/Authority/Merge.t broken in 34739 -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34739 --- Comment #33 from Janusz Kaczmarek <januszop@gmail.com> --- My fault :( Thanks, Thomas! -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34739 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to main |Needs documenting CC| |fridolin.somers@biblibre.co | |m --- Comment #35 from Fridolin Somers <fridolin.somers@biblibre.com> --- Looks great but I prefer not impact 24.11.x LTS in case of unknown side-effects. -- You are receiving this mail because: You are watching all bug changes.