https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 --- Comment #1 from David Gustafsson <glasklas@gmail.com> --- Created attachment 105065 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=105065&action=edit Bug 25539: Remove AddBiblio option "defer_marc_save" Items are no longer embedded in the MARCXML and because of this the MARC data does not need to be saved once more after changing record items data. The "defer_marc_save" is no longer needed since bulkmarcimport.pl was the only place this option was utilized in order to resave MARC data after possibly changing items data. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |14957 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=14957 [Bug 14957] Write protecting MARC fields based on source of import -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version|unspecified |master -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |glasklas@gmail.com |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 --- Comment #4 from David Gustafsson <glasklas@gmail.com> --- Yes, I usually include a test-plan, this fix will not produce any change of behaviour with the current code, but causes a bug together with bug 14957. So in that case the test-plan would have to include applying that patch. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |martin.renvoize@ptfs-europe | |.com --- Comment #6 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to David Gustafsson from comment #4)

Yes, I usually include a test-plan, this fix will not produce any change of behaviour with the current code, but causes a bug together with bug 14957. So in that case the test-plan would have to include applying that patch.

No change in behaviour can still be tested with and without the patch. I think it would really help to get this moving and unlock bug 14957. But looking at the patch I get an idea of why this is hard. Some example uses might still be good to note and test. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off --- Comment #8 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- It took a little getting my head around, but I believe this is all correct. For reference, the commit adding the defer_marc_save option had the message: ==== item rework: replaced AddBiblioAndItems Replace C4::Biblio::AddBiblioAndItems with two things: * An option to C4::Biblio::AddBiblio to defer writing biblioitems.marc and biblioitems.marcxml. This option was created to give a significant speed boost to bulkmarcimport.pl, but is *not* recommended for general use. * C4::Items::AddItemBatchFromMarc This refactoring removes the need to have functions in C4::Biblio and C4::Items that call each other's private functions. Signed-off-by: Chris Cormack <crc@liblime.com> Signed-off-by: Joshua Ferraro <jmf@liblime.com> ==== I believe it is no defunk as David suggests. I tried a few runs of bulkmarcimport with and without the patch and found no really noticeable slowdown in performance (though performance of this script really isn't very good these days :( ) Signing off. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 --- Comment #10 from David Gustafsson <glasklas@gmail.com> --- I had a look at this, and I think this is a bit of a mess to sort out. You are calling bulkmarcimport.pl without -update or -insert set, which seems to default to -insert ($insert = 1, $update = undef), and without -match with a matchpoint. bulkmarcimport will correctly get original id through: $originalid = GetRecordId( $record, $tagid, $subfieldid ); But will not use $originalid as a match ($biblionumber will have no value on line 453 where record is either inserted or updated). Because of this and $update = undef the block inside of if ($insert) {... on line 470 will run inserting a new record with defer_marc_save => 1. After this block: eval { ( $itemnumbers_ref, $errors_ref ) = AddItemBatchFromMarc( $record, $biblionumber, $biblioitemnumber, '' ); }; Will be run with a $biblionumber set to a biblio without any marc data yet saved. Inside of AddItemBatchFromMarc my $item_object = Koha::Item->new($item)->store; will be run. Inside Koha/Item.pm Koha::Item::store this will get executed: C4::Biblio::ModZebra( $self->biblionumber, "specialUpdate", "biblioserver" ) unless $params->{skip_modzebra_update} ModZebra will attempt to load the Biblio Marc data: $record = GetMarcBiblio({ biblionumber => $biblionumber, embed_items => 1 }); But this will return an undefined record since the marc-data has not yet been saved. This will cause the import in later processing of this undefined $record to crash which will not propagate to bulkmarcimport.pl since run within an eval. I get this when running bulkmarcimport without the patch. The new record marc data is later saved in the following block which comes shortly after: my $clone_record = $record->clone(); C4::Biblio::_strip_item_fields($clone_record, ''); # This sets the marc fields if there was an error, and also calls # defer_marc_save. ModBiblioMarc( $clone_record, $biblionumber, $framework ); which I removed in the patch (was not aware that the marc data was loaded as a side effect of AddItemBarchFromMarc). This code needs to be removed somehow though since it will bypass the marc merge rules in bug 14957. I think by removing this block a new bug is exposed which is that AddBiblio will not strip items before saving. Perhaps this could be addressed by stripping items before calling AddBiblio in bulkmarcimport.pl. I will look further into this. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 --- Comment #11 from David Gustafsson <glasklas@gmail.com> --- Created attachment 110260 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=110260&action=edit Strip items when adding new biblio to preserve previous behaviour of the script -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 --- Comment #13 from David Gustafsson <glasklas@gmail.com> --- I have now modified the patch so that items will be stripped also when adding new biblios. You will still get an error message when adding an exported biblio with existing items since the item ids will cause a duplicate primary key constraint error. If understanding this correctly this should be considered "better" or in line with the current behavior of the script since the unmodified script will bail out even earlier with a different error (triggered by not saving the marc when attempting to load it). Even if the current script would get past that (which it would if removing the "defer_marc_save" option) you would still get the exact same error since AddItemBatchFromMarc would be called on the $record with updated internal bibio ids but with the same item-ids. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 --- Comment #14 from David Gustafsson <glasklas@gmail.com> --- About the whitespace in the patch, I removed superfluous white space in some places, but sure, to beconsistent that should perhaps be another issue since there is also some other instances of tabs instead of spaces and other incorrect formatting that could warrant another issue. I can remove this from the patch in this case. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks|14957 | Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=14957 [Bug 14957] Write protecting MARC fields based on source of import -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 --- Comment #17 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- I commented out the ability to use overlay rules with the bulkmarcimport route.. that unblocks that bug so it can be pushed and allows us to continue work here to re-enable that part of the feature. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 Frank Hansen <frank.hansen@ub.lu.se> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |frank.hansen@ub.lu.se -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 --- Comment #19 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- (In reply to Victor Grousset/tuxayo from comment #18)

...

Removing bug 14957 as a dependant

Actually it was re-added ^^

Actually, it was a swap ;).. this was blocking the other bug, now it depends upon it ;) -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 --- Comment #21 from David Gustafsson <glasklas@gmail.com> --- I completely forgot to add why I brought the patch up. The reason being that it's hard to fix the issue without exposing other bugs as mentioned in comment #13. It would probably be better to perform a more thorough refactor, but then there is the issue that we already did that but does not have a clean patch-set to contribute. Perhaps I will have to take the time to go through our current patch and verify that the current options are still supported and the script behaves in more or less the same way, so no or minimal API breakage. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 --- Comment #23 from David Gustafsson <glasklas@gmail.com> --- Sorry about the late reply, I posted the commit in comment #20, not sure it applies to the current Koha master, we regularly rebase our patch-set but have not done so in a while. Even though I think it could be useful to have a look at I hope to work more on cleaning this mess up, but not sure when will able to do so. Some changes might be easier to break out than others. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks|14957 | Depends on| |14957 CC| |jonathan.druart+koha@gmail. | |com Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=14957 [Bug 14957] Write protecting MARC fields based on source of import -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on|14957 |29440 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=14957 [Bug 14957] Write protecting MARC fields based on source of import https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 [Bug 29440] Refactor/clean up bulkmarcimport.pl -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 --- Comment #25 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- That's great news David, thankyou for getting back onto this.. Do feel free to grab me to take a look when you need review etc :) -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 --- Comment #27 from David Gustafsson <glasklas@gmail.com> --- Created attachment 127599 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=127599&action=edit Bug 25539: Remove AddBiblio option "defer_marc_save" Items are no longer embedded in the MARCXML and because of this the MARC data does not need to be saved once more after changing record items data. The "defer_marc_save" is no longer needed since bulkmarcimport.pl was the only place this option was utilized in order to resave MARC data after possibly changing items data. There is also a bug bulkmarcimport.pl where the record data is re-saved without stripping items if duplicate items are found and the dedup barcodes option is enabled that is resolved by this change. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #109789|0 |1 is obsolete| | Attachment #127599|0 |1 is obsolete| | --- Comment #29 from David Gustafsson <glasklas@gmail.com> --- Created attachment 127601 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=127601&action=edit Bug 25539: Remove AddBiblio option "defer_marc_save" Items are no longer embedded in the MARCXML and because of this the MARC data does not need to be saved once more after changing record items data. The "defer_marc_save" is no longer needed since bulkmarcimport.pl was the only place this option was utilized in order to resave MARC data after possibly changing items data. There is also a bug bulkmarcimport.pl where the record data is re-saved without stripping items if duplicate items are found and the dedup barcodes option is enabled that is resolved by this change. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 --- Comment #31 from David Gustafsson <glasklas@gmail.com> --- Now rebased upon bug 29440 -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=14957 -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #127601|0 |1 is obsolete| | --- Comment #33 from David Gustafsson <glasklas@gmail.com> --- Created attachment 127976 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=127976&action=edit Bug 25539: Remove AddBiblio option "defer_marc_save" Items are no longer embedded in the MARCXML and because of this the MARC data does not need to be saved once more after changing record items data. The "defer_marc_save" is no longer needed since bulkmarcimport.pl was the only place this option was utilized in order to resave MARC data after possibly changing items data. There is also a bug bulkmarcimport.pl where the record data is re-saved without stripping items if duplicate items are found and the dedup barcodes option is enabled that is resolved by this change. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #127976|0 |1 is obsolete| | --- Comment #35 from David Gustafsson <glasklas@gmail.com> --- Created attachment 127978 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=127978&action=edit Bug 25539: Remove AddBiblio option "defer_marc_save" Items are no longer embedded in the MARCXML and because of this the MARC data does not need to be saved once more after changing record items data. The "defer_marc_save" is no longer needed since bulkmarcimport.pl was the only place this option was utilized in order to resave MARC data after possibly changing items data. There is also a bug bulkmarcimport.pl where the record data is re-saved without stripping items if duplicate items are found and the dedup barcodes option is enabled that is resolved by this change. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #127978|0 |1 is obsolete| | --- Comment #37 from David Gustafsson <glasklas@gmail.com> --- Created attachment 127980 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=127980&action=edit Bug 25539: Remove AddBiblio option "defer_marc_save" Items are no longer embedded in the MARCXML and because of this the MARC data does not need to be saved once more after changing record items data. The "defer_marc_save" is no longer needed since bulkmarcimport.pl was the only place this option was utilized in order to resave MARC data after possibly changing items data. There is also a bug bulkmarcimport.pl where the record data is re-saved without stripping items if duplicate items are found and the dedup barcodes option is enabled that is resolved by this change. This change enables MARC Overlay rules to be enabled for bulkmarkcimport.pl as using the defer_marc option would previously effectively bypass any defined rules. To test 1) Apply patch 29440 (which this depends on), but no not yet apply this patch 2) Enable MARC overlay rules and add a rule (with source => "*" or 'bulkmarcimport') to protect some field. 3) Import a biblio using the bulkmarcimport script. Overlay rules will only be applied on updates so a match condition need to be supplied (for example -match "control-number,001") 4) Edit the imported biblio and change the value of the protected field 5) Run the import again and verify the the field that should have been protected has been overwritten 6) Apply the patch and repeat steps 4-5, verify that the field was protected and has not been overwritten -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 --- Comment #39 from Frank Hansen <frank.hansen@ub.lu.se> --- (In reply to David Gustafsson from comment #37)

Created attachment 127980 [details] [review] Bug 25539: Remove AddBiblio option "defer_marc_save"

2) Enable MARC overlay rules and add a rule (with source => "*" or 'bulkmarcimport') to protect some field.

Hi David. It is not possible to configure bulkmarcimport as source filter in home->administration->Marc overlay rules because bulkmarcimport.pl is commented out in koha-tmpl/intranet-tmpl/prog/en/modules/admin/marc-overlay-rules.tt template. Is it commented out by intention? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #127980|0 |1 is obsolete| | --- Comment #41 from David Gustafsson <glasklas@gmail.com> --- Created attachment 133030 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=133030&action=edit Bug 25539: Remove AddBiblio option "defer_marc_save" Items are no longer embedded in the MARCXML and because of this the MARC data does not need to be saved once more after changing record items data. The "defer_marc_save" is no longer needed since bulkmarcimport.pl was the only place this option was utilized in order to resave MARC data after possibly changing items data. There is also a bug bulkmarcimport.pl where the record data is re-saved without stripping items if duplicate items are found and the dedup barcodes option is enabled that is resolved by this change. This change enables MARC Overlay rules to be enabled for bulkmarkcimport.pl as using the defer_marc option would previously effectively bypass any defined rules. To test 1) Apply patch 29440 (which this depends on), but no not yet apply this patch 2) Remove comments around "overlay_context => { source => 'bulkmarcimport' }" in misc/migration_tools/bulkmarcimport.pl line 129 3) Enable MARC overlay rules and add a rule (with source => "*" or 'bulkmarcimport') to protect some field. 4) Import a biblio using the bulkmarcimport script. Overlay rules will only be applied on updates so a match condition matching the record in Koha when bulkmarcimport.pl is run the next time for the same record needs to be supplied (for example -match "control-number,001") 5) Edit the imported biblio and change the value of the protected field 6) Run bulkmarcimport with the same parameters as in 4) and verify the the field that should have been protected has been overwritten 7) Revert changes for line 129 in bulkmarcimport.pl (or will result in conflict applying the patch), apply the patch and repeat steps 4-5, verify that the field now was protected from being overwritten. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 --- Comment #43 from David Gustafsson <glasklas@gmail.com> --- Created attachment 133032 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=133032&action=edit Bug 25539: Enable bulkmarcimport overlay context -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #133030|0 |1 is obsolete| | --- Comment #45 from David Gustafsson <glasklas@gmail.com> --- Created attachment 133034 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=133034&action=edit Bug 25539: Remove AddBiblio option "defer_marc_save" Items are no longer embedded in the MARCXML and because of this the MARC data does not need to be saved once more after changing record items data. The "defer_marc_save" is no longer needed since bulkmarcimport.pl was the only place this option was utilized in order to resave MARC data after possibly changing items data. There is also a bug bulkmarcimport.pl where the record data is re-saved without stripping items if duplicate items are found and the dedup barcodes option is enabled that is resolved by this change. This change enables MARC Overlay rules to be enabled for bulkmarkcimport.pl as using the defer_marc option would previously effectively bypass any defined rules. To test 1) Apply patch 29440 (which this depends on), but no not yet apply this patch 2) Remove comments around "bulkmarcimport: _("bulkmarcimport.pl")," in koha-tmpl/intranet-tmpl/prog/en/modules/admin/marc-overlay-rules.tt line 463 3) Enable MARC overlay rules and add a rule (with source => "*" or 'bulkmarcimport') to protect some field. 4) Import a biblio using the bulkmarcimport script. Overlay rules will only be applied on updates so a match condition matching the record in Koha when bulkmarcimport.pl is run the next time for the same record needs to be supplied (for example -match "control-number,001") 5) Edit the imported biblio and change the value of the protected field 6) Run bulkmarcimport with the same parameters as in 4) and verify the the field that should have been protected has been overwritten 7) Revert changes for line 463 in marc-overlay-rules.tt (or will result in conflict applying the patch), apply the patch and repeat steps 4-5, verify that the field now was protected from being overwritten. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #133032|0 |1 is obsolete| | --- Comment #47 from David Gustafsson <glasklas@gmail.com> --- Created attachment 133036 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=133036&action=edit Bug 25539: Enable bulkmarcimport overlay context -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #133034|0 |1 is obsolete| | --- Comment #49 from David Gustafsson <glasklas@gmail.com> --- Created attachment 140992 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=140992&action=edit Bug 25539: Remove AddBiblio option "defer_marc_save" Items are no longer embedded in the MARCXML and because of this the MARC data does not need to be saved once more after changing record items data. The "defer_marc_save" is no longer needed since bulkmarcimport.pl was the only place this option was utilized in order to resave MARC data after possibly changing items data. There is also a bug bulkmarcimport.pl where the record data is re-saved without stripping items if duplicate items are found and the dedup barcodes option is enabled that is resolved by this change. This change enables MARC Overlay rules to be enabled for bulkmarkcimport.pl as using the defer_marc option would previously effectively bypass any defined rules. To test 1) Apply patch 29440 (which this depends on), but no not yet apply this patch 2) Remove comments around "bulkmarcimport: _("bulkmarcimport.pl")," in koha-tmpl/intranet-tmpl/prog/en/modules/admin/marc-overlay-rules.tt line 463 3) Enable MARC overlay rules and add a rule (with source => "*" or 'bulkmarcimport') to protect some field. 4) Import a biblio using the bulkmarcimport script. Overlay rules will only be applied on updates so a match condition matching the record in Koha when bulkmarcimport.pl is run the next time for the same record needs to be supplied (for example -match "control-number,001") 5) Edit the imported biblio and change the value of the protected field 6) Run bulkmarcimport with the same parameters as in 4) and verify the the field that should have been protected has been overwritten 7) Revert changes for line 463 in marc-overlay-rules.tt (or will result in conflict applying the patch), apply the patch and repeat steps 4-5, verify that the field now was protected from being overwritten. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #133036|0 |1 is obsolete| | --- Comment #51 from David Gustafsson <glasklas@gmail.com> --- Created attachment 140994 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=140994&action=edit Bug 25539: Enable bulkmarcimport overlay context -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 Magnus Enger <magnus@libriotech.no> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |magnus@libriotech.no Status|Needs Signoff |BLOCKED --- Comment #53 from Magnus Enger <magnus@libriotech.no> --- Depends on Bug 29440, which currently does not apply. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #140993|0 |1 is obsolete| | --- Comment #55 from David Gustafsson <glasklas@gmail.com> --- Created attachment 156225 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=156225&action=edit Bug 25539: Strip items when adding new biblio to preserve previous behaviour of the script -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 --- Comment #57 from David Gustafsson <glasklas@gmail.com> --- Rebased against master. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 M <schodkowy.omegi-0r@icloud.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |35075 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35075 [Bug 35075] "bulkmarcimport" source missing in Marc Overlay Rules settings page -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 --- Comment #59 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to M from comment #58)

...

7) Revert changes for line 463 in marc-overlay-rules.tt (or will result in conflict applying the patch), apply the patch and repeat steps 4-5, verify that the field now was protected from being overwritten.

Generally it works, but doesn't for field 942 for some reason... It just gets removed, while others like 866 are preserved (with exact same overlay rules...). Perhaps related to Bug 34191 (meaning that it might be unrelated to this patchset)

Don't use the asterisk - then it should work. See: Bug 33268 - Overlay rules don't work correctly when source is set to * But there should be no difference for different fields. Do you have a second rule for 942 by chance? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 --- Comment #61 from M <schodkowy.omegi-0r@icloud.com> --- I ran another test and it does seem to be the case somehow still. Original record in Koha has fields 866 and 942 (the latter is Koha metadata, including required item type). The imported record has both of these fields missing. Both of these fields have exact same overlay rules setup up, the field ID being the only different thing between them. I did a dirty patch to the script to copy over field 942 from found record for matching, and only that seems to help... Weird thing, it'd be nice if someone else could try to reproduce this situation, maybe I'm doing something wrong, but I can't seem to spot what's the difference either... -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 --- Comment #63 from M <schodkowy.omegi-0r@icloud.com> --- "id","tag","module","filter","add","append","remove","delete" "1","942","source","bulkmarcimport","1","1","0","0" "2","967","source","bulkmarcimport","0","0","0","0" "3","852","source","bulkmarcimport","1","1","0","0" "4","866","source","bulkmarcimport","1","1","0","0" "6","003","source","bulkmarcimport","1","0","0","0" I initially had add+append set to 0, and I think setting them to 1 like 8xx was one of attempts at finding the root cause. Last few attempts were done with this exact settings, after changing the filter back to be the same. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 --- Comment #65 from M <schodkowy.omegi-0r@icloud.com> --- Sorry for the fuss... cache! The problem was cache. Some kind of cache was holding the old marc overlay rules, after restarting and flushing everything the changed rules are now correctly applied. Now everything seems to work perfectly fine. I'm not gonna lie, I share a huge hope this changeset could be merged in for 23.11 release in such case so that we can use those scripts for daily upstream updates -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 --- Comment #67 from M <schodkowy.omegi-0r@icloud.com> ---

But only bug fixes will usually be ported back - so it might be too late for 23.11 now.

Eh, I daresay it'd fall under big fixes technically, since it fixes the overlay rules application on imported records, and I understand that it was one of the main points of the script's refactor to have that working properly as a result. It's possible that actually the dependency fixes that bug, but this patch here re-enables the overlay context (also both made by the same author, so I assume it's some kind of logical continuation) -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 M <schodkowy.omegi-0r@icloud.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #156224|0 |1 is obsolete| | --- Comment #69 from M <schodkowy.omegi-0r@icloud.com> --- Created attachment 160158 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=160158&action=edit Bug 25539: Remove AddBiblio option "defer_marc_save" -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 M <schodkowy.omegi-0r@icloud.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #156226|0 |1 is obsolete| | --- Comment #71 from M <schodkowy.omegi-0r@icloud.com> --- Created attachment 160160 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=160160&action=edit Bug 25539: Enable bulkmarcimport overlay context -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 --- Comment #73 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- This has been on my radar for a long time. I'll jump on it first thing in the morning and get to through QA now it rebases and signed off again. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #160158|0 |1 is obsolete| | --- Comment #75 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 160489 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=160489&action=edit Bug 25539: Remove AddBiblio option "defer_marc_save" Items are no longer embedded in the MARCXML and because of this the MARC data does not need to be saved once more after changing record items data. The "defer_marc_save" is no longer needed since bulkmarcimport.pl was the only place this option was utilized in order to resave MARC data after possibly changing items data. There is also a bug bulkmarcimport.pl where the record data is re-saved without stripping items if duplicate items are found and the dedup barcodes option is enabled that is resolved by this change. This change enables MARC Overlay rules to be enabled for bulkmarkcimport.pl as using the defer_marc option would previously effectively bypass any defined rules. To test 1) Apply patch 29440 (which this depends on), but no not yet apply this patch 2) Remove comments around "bulkmarcimport: _("bulkmarcimport.pl")," in koha-tmpl/intranet-tmpl/prog/en/modules/admin/marc-overlay-rules.tt line 463 3) Enable MARC overlay rules and add a rule (with source => "*" or 'bulkmarcimport') to protect some field. 4) Import a biblio using the bulkmarcimport script. Overlay rules will only be applied on updates so a match condition matching the record in Koha when bulkmarcimport.pl is run the next time for the same record needs to be supplied (for example -match "control-number,001") 5) Edit the imported biblio and change the value of the protected field 6) Run bulkmarcimport with the same parameters as in 4) and verify the the field that should have been protected has been overwritten 7) Revert changes for line 463 in marc-overlay-rules.tt (or will result in conflict applying the patch), apply the patch and repeat steps 4-5, verify that the field now was protected from being overwritten. Signed-off-by: Michał Kula <148193449+mkibp@users.noreply.github.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #160160|0 |1 is obsolete| | --- Comment #77 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 160491 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=160491&action=edit Bug 25539: Enable bulkmarcimport overlay context Signed-off-by: Michał Kula <148193449+mkibp@users.noreply.github.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 --- Comment #78 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- *** Bug 35075 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 --- Comment #79 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Hm, a bugfix depending on an enhancement can be an issue as it makes backporting hard or impossible. Any chance we could get a bugfix for the stable versions independent of bug 29440? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to master Version(s)| |24.05.00 released in| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED Status|Pushed to master |RESOLVED CC| |fridolin.somers@biblibre.co | |m --- Comment #82 from Fridolin Somers <fridolin.somers@biblibre.com> --- Depends on Bug 29440 not in 23.11.x -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25539 --- Comment #84 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Fridolin Somers from comment #83)

Note there is an error in commit to master : b374358e84 Bug 35539: (QA follow-up) Remove defer_marc_save from Elasticsearch.t

Maybe need to fix manually in release notes.

Thanks Frido - I think it was bound to happen. -- You are receiving this mail because: You are watching all bug changes.