https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 --- Comment #1 from David Gustafsson <glasklas@gmail.com> --- Created attachment 127479 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=127479&action=edit Bug 29440: Make authorities import behavior consistent with biblios The current behavior for when importing authorities, when match option is enabled, is to insert as a new authority fetching from db fails. This is both a bit confusing and not consistent with the biblio case where an error is generated. Also clean up error and log messages so both messages/errors and when messages/errors are issued are more consistent for the biblio and authority case. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #127479|0 |1 is obsolete| | --- Comment #3 from David Gustafsson <glasklas@gmail.com> --- Created attachment 127481 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=127481&action=edit Bug 29440: Make authorities import behavior consistent with biblios The current behavior for when importing authorities, when match option is enabled, is to insert as a new authority fetching from db fails. This is both a bit confusing and not consistent with the biblio case where an error is generated. Also clean up error and log messages so both messages/errors and when messages/errors are issued are more consistent for the biblio and authority case. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 --- Comment #4 from David Gustafsson <glasklas@gmail.com> --- To make this easier to review I will make a rough summary/motivation of the major changes, just did not have the time to do so right now. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #127481|0 |1 is obsolete| | --- Comment #6 from David Gustafsson <glasklas@gmail.com> --- Created attachment 127486 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=127486&action=edit Bug 29440: Make authorities import behavior consistent with biblios The current behavior for when importing authorities, when match option is enabled, is to insert as a new authority fetching from db fails. This is both a bit confusing and not consistent with the biblio case where an error is generated. Also clean up error and log messages so both messages/errors and when messages/errors are issued are more consistent for the biblio and authority case. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 --- Comment #8 from David Gustafsson <glasklas@gmail.com> --- Yes, I will make sure to mark the other patches (will probably only be one or two more besides 25539) as depending on this one, as far as I understand this is all I have to do to make this an omnibus bug? For reviewers of this patch, here is a short summary of the changes: Added an option 'defer_search_engine_indexing' to ModBiblio/Modbiblio marc, which is used in the script for avoiding indexing each item individually and to do so in bulk when changes are committed to the database. This is more efficient, and you don't run the same risk of ending up with junk data in the index if the script crashes and the current database transaction is rolled back. Instead of processing records in one loop, the processing is split up into two loops. The first one for pre-processing tasks (transcoding, filtering out broken records etc) that does not touch the database. This makes the code in the import loop cleaner, and it's also nice to have the option of bailing out before any database changes are committed (could for example add the option to abort the whole import if contains one or more invalid records). The rest of the changes more or less of fixing formatting/indentation, renaming some variables for clarity purposes, making error/log messages more consistent and generally just cleaning up the code. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #127486|0 |1 is obsolete| | --- Comment #10 from David Gustafsson <glasklas@gmail.com> --- Created attachment 127509 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=127509&action=edit Bug 29440: Make authorities import behavior consistent with biblios The current behavior for when importing authorities, when match option is enabled, is to insert as a new authority fetching from db fails. This is both a bit confusing and not consistent with the biblio case where an error is generated. Also clean up error and log messages so both messages/errors and when messages/errors are issued are more consistent for the biblio and authority case. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #127509|0 |1 is obsolete| | --- Comment #12 from David Gustafsson <glasklas@gmail.com> --- Created attachment 127652 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=127652&action=edit Bug 29440: Make authorities import behavior consistent with biblios The current behavior for when importing authorities, when match option is enabled, is to insert as a new authority fetching from db fails. This is both a bit confusing and not consistent with the biblio case where an error is generated. Also clean up error and log messages so both messages/errors and when messages/errors are issued are more consistent for the biblio and authority case. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #127652|0 |1 is obsolete| | --- Comment #14 from David Gustafsson <glasklas@gmail.com> --- Created attachment 127792 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=127792&action=edit Bug 29440: Make authorities import behavior consistent with biblios The current behavior for when importing authorities, when match option is enabled, is to insert as a new authority fetching from db fails. This is both a bit confusing and not consistent with the biblio case where an error is generated. Also clean up error and log messages so both messages/errors and when messages/errors are issued are more consistent for the biblio and authority case. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #127792|0 |1 is obsolete| | --- Comment #16 from David Gustafsson <glasklas@gmail.com> --- Created attachment 127796 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=127796&action=edit Bug 29440: Make authorities import behavior consistent with biblios The current behavior for when importing authorities, when match option is enabled, is to insert as a new authority fetching from db fails. This is both a bit confusing and not consistent with the biblio case where an error is generated. Also clean up error and log messages so both messages/errors and when messages/errors are issued are more consistent for the biblio and authority case. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 --- Comment #18 from David Gustafsson <glasklas@gmail.com> --- Hi! As it wasn't really ready for signoff I did not change the status. Have done some manual testing to go through most of the options and catch possible issues after splitting the code into feature branches. There are no (serious at least) bugs fixed, and no features added, just a clean up, so what need to make sure is that the script works as before. There is no test-suite for bulkmarkimport.pl so it's a lot of work for reviewers to go through all the options and fully test the script, but I can at least provide some basic steps wit the most common options. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #127796|0 |1 is obsolete| | --- Comment #20 from David Gustafsson <glasklas@gmail.com> --- Created attachment 127960 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=127960&action=edit Bug 29440: Make authorities import behavior consistent with biblios The current behavior for when importing authorities, when match option is enabled, is to insert as a new authority fetching from db fails. This is both a bit confusing and not consistent with the biblio case where an error is generated. Also clean up error and log messages so both messages/errors and when messages/errors are issued are more consistent for the biblio and authority case. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #127961|Authorities MARCXML example |authorities.xml description| | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bernard.scaife@ptfs-europe. | |com -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #127959|0 |1 is obsolete| | --- Comment #24 from David Gustafsson <glasklas@gmail.com> --- Created attachment 128086 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=128086&action=edit Bug 29440: Refactor and clean up bulkmarcimport.pl To test: 1) Import biblios using ./bulkmarcimport.pl -v -b -file biblios.mrc -insert -update -c=MARC21 -match "control-number,001" -n=1 -l "/tmp/import.log" 2) Verify in /tmp/import.log that one biblio has been inserted 3) Import again without the "-n" option, verify that one biblio has been updated and the rest inserted 4) In the staff interface search the catalog for some string appearing in the imported biblios to verify records have been indexed 5) View/edit some of the biblios the staff interface to very have been properly imported 6) Import authorities using ./bulkmarcimport.pl -v -m=MARCXML -a -file authorities.xml -insert -update -c=MARC21 -keepids="024a" -match "heading-main,100a" -l "/tmp/import.log" -yaml="/tmp/ids.yml" 7) Go to the authorities page and find the imported authority by for example search for a 400 or 100 field value. Verify that the 001 field of the incoming record has been moved to 024a. Also check that the value of 024a appear as a key in /tmp/ids.yml. 8) Edit the authorities.xml file and change the value of field 400a 9) Import authorities again using the same options 10) Verify that the 400a field of the authority in Koha has not been updated (the import was skippped since revision did not change) 11) Edit authorities.xml, replace the value of field 005 in the file with field 005 of the authority in Koha incremented by one 12) Import again with the same options 13) Verify that the 400a field of the authority in Koha now have been updated -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 --- Comment #26 from David Gustafsson <glasklas@gmail.com> --- (In reply to Martin Renvoize from comment #23)

Nice work David..

On with the review: 1. You appear to introduce a few new modules, but I'm not seeing some of them actually used. [List::MoreUtils qw(any), Koha::Plugins::Handler;] 2. With such an extensive tidy up.. I'd probably go ahead and run it through perltidy as a final patch.. (I did this on the before and after script to make comparing simpler)

Otherwise, this is generally looking very reasonable to me.

I'm going to run a few more test runs against it and ask our migrations team to have a play

Thanks. Yes those imports where part of add features I removed (but forgot to remove the imports). I'm not a regular user of perltidy, but for me the output-file is the same as the original file. Perhaps I'm doing something wrong. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 --- Comment #27 from David Gustafsson <glasklas@gmail.com> --- Discovered looks like there is encoding issues with "biblios.mrc", double encoded most likely, I suspect the file was re-encoded either on download/upload as the original file correctly encoded. If requested I can upload another file and tar it beforehand. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #127963|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |glasklas@gmail.com |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 Frank Hansen <frank.hansen@ub.lu.se> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 Frank Hansen <frank.hansen@ub.lu.se> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #128087|0 |1 is obsolete| | --- Comment #30 from Frank Hansen <frank.hansen@ub.lu.se> --- Created attachment 128891 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=128891&action=edit Bug 29440: Make authorities import behavior consistent with biblios The current behavior for when importing authorities, when match option is enabled, is to insert as a new authority fetching from db fails. This is both a bit confusing and not consistent with the biblio case where an error is generated. Also clean up error and log messages so both messages/errors and when messages/errors are issued are more consistent for the biblio and authority case. Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se> The patch works fine for me. Signing off. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #128890|0 |1 is obsolete| | --- Comment #31 from David Gustafsson <glasklas@gmail.com> --- Created attachment 131176 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=131176&action=edit Bug 29440: Refactor and clean up bulkmarcimport.pl To test: 1) Import biblios using ./bulkmarcimport.pl -v -b -file biblios.mrc -insert -update -c=MARC21 -match "control-number,001" -n=1 -l "/tmp/import.log" 2) Verify in /tmp/import.log that one biblio has been inserted 3) Import again without the "-n" option, verify that one biblio has been updated and the rest inserted 4) In the staff interface search the catalog for some string appearing in the imported biblios to verify records have been indexed 5) View/edit some of the biblios the staff interface to very have been properly imported 6) Import authorities using ./bulkmarcimport.pl -v -m=MARCXML -a -file authorities.xml -insert -update -c=MARC21 -keepids="024a" -match "heading-main,100a" -l "/tmp/import.log" -yaml="/tmp/ids.yml" 7) Go to the authorities page and find the imported authority by for example search for a 400 or 100 field value. Verify that the 001 field of the incoming record has been moved to 024a. Also check that the value of 024a appear as a key in /tmp/ids.yml. 8) Edit the authorities.xml file and change the value of field 400a 9) Import authorities again using the same options 10) Verify that the 400a field of the authority in Koha has not been updated (the import was skippped since revision did not change) 11) Edit authorities.xml, replace the value of field 005 in the file with field 005 of the authority in Koha incremented by one 12) Import again with the same options 13) Verify that the 400a field of the authority in Koha now have been updated -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 --- Comment #33 from David Gustafsson <glasklas@gmail.com> --- Rebased against master -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #131177|0 |1 is obsolete| | --- Comment #35 from David Gustafsson <glasklas@gmail.com> --- Created attachment 131181 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=131181&action=edit Bug 29440: Make authorities import behavior consistent with biblios The current behavior for when importing authorities, when match option is enabled, is to insert as a new authority fetching from db fails. This is both a bit confusing and not consistent with the biblio case where an error is generated. Also clean up error and log messages so both messages/errors and when messages/errors are issued are more consistent for the biblio and authority case. Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se> The patch works fine for me. Signing off. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 --- Comment #37 from David Gustafsson <glasklas@gmail.com> --- Created attachment 132942 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132942&action=edit Bug 29440: Fix QA-tool issues -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=30822 -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |30822 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30822 [Bug 30822] BatchCommit does not deal with indexation correctly -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 --- Comment #40 from David Gustafsson <glasklas@gmail.com> --- Created attachment 140822 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=140822&action=edit Bug 29440: defer biblio autolinking until records are indexed -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 --- Comment #42 from David Gustafsson <glasklas@gmail.com> --- (In reply to Martin Renvoize from comment #38)

Looking good so far.. a couple of comments

We have a fairly standard 'skip_index' name for the search engine index skip param.. we should stick to that here too (I bet your patch pre-dates that, but it shouldn't be too onerous to update for consistency)

However, there's a more challenging thing to resolve.

If 'BiblioAddsAuthorities' is enabled the index skip is going to cause issues as the linker relies of updated indexes to do it's job... we possibly need to move where the BiblioAddsAuthories code runs for such cases to run at the end after a first run of indexing has taken place and then run a second index run to catch the link changes it makes?

Conundrum.

Good catch. I'm a little bit confused about the first point. I only see skip_index being used in rebuild_zebra.pl and not really sure what it does. bulkmarkimport.pl will always defer indexing (but only for elasticsearch) and bulk-index for each batch. There is not option to disable this as there is no apparent downside not to do this. The second issue should now be fixed, it turned out to be pretty straightforward as only needed to add the option to skip linking for AddBiblio (the option was already present in ModBiblio), and then run BiblioAutoLink on all records in the batch after indexing them successfully (if I'm not missing anything). The patch is still not rebased but will now have a look at that. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #131181|0 |1 is obsolete| | --- Comment #44 from David Gustafsson <glasklas@gmail.com> --- Created attachment 140904 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=140904&action=edit Bug 29440: Make authorities import behavior consistent with biblios The current behavior for when importing authorities, when match option is enabled, is to insert as a new authority fetching from db fails. This is both a bit confusing and not consistent with the biblio case where an error is generated. Also clean up error and log messages so both messages/errors and when messages/errors are issued are more consistent for the biblio and authority case. Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se> The patch works fine for me. Signing off. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #140823|0 |1 is obsolete| | --- Comment #46 from David Gustafsson <glasklas@gmail.com> --- Created attachment 140906 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=140906&action=edit Bug 29440: defer biblio autolinking until records are indexed -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 --- Comment #48 from David Gustafsson <glasklas@gmail.com> --- Typo: Should have been "Have *now* rebased against master". -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #140903|0 |1 is obsolete| | --- Comment #50 from David Gustafsson <glasklas@gmail.com> --- Created attachment 140924 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=140924&action=edit Bug 29440: Refactor and clean up bulkmarcimport.pl To test: 1) Import biblios using ./bulkmarcimport.pl -v -b -file biblios.mrc -insert -update -c=MARC21 -match "control-number,001" -n=1 -l "/tmp/import.log" 2) Verify in /tmp/import.log that one biblio has been inserted 3) Import again without the "-n" option, verify that one biblio has been updated and the rest inserted 4) In the staff interface search the catalog for some string appearing in the imported biblios to verify records have been indexed 5) View/edit some of the biblios the staff interface to very have been properly imported 6) Import authorities using ./bulkmarcimport.pl -v -m=MARCXML -a -file authorities.xml -insert -update -c=MARC21 -keepids="024a" -match "heading-main,100a" -l "/tmp/import.log" -yaml="/tmp/ids.yml" 7) Go to the authorities page and find the imported authority by for example search for a 400 or 100 field value. Verify that the 001 field of the incoming record has been moved to 024a. Also check that the value of 024a appear as a key in /tmp/ids.yml. 8) Edit the authorities.xml file and change the value of field 400a 9) Import authorities again using the same options 10) Verify that the 400a field of the authority in Koha has not been updated (the import was skippped since revision did not change) 11) Edit authorities.xml, replace the value of field 005 in the file with field 005 of the authority in Koha incremented by one 12) Import again with the same options 13) Verify that the 400a field of the authority in Koha now have been updated -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #140905|0 |1 is obsolete| | --- Comment #52 from David Gustafsson <glasklas@gmail.com> --- Created attachment 140926 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=140926&action=edit Bug 29440: Fix QA-tool issues -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 --- Comment #54 from David Gustafsson <glasklas@gmail.com> --- Created attachment 140928 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=140928&action=edit Bug 29440: replace usage of removed GetMarcBiblio -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 --- Comment #56 from David Gustafsson <glasklas@gmail.com> --- Created attachment 140987 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=140987&action=edit Bug 29440: Remove pointless call to _strip_item_fields -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 --- Comment #58 from David Gustafsson <glasklas@gmail.com> --- Created attachment 140989 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=140989&action=edit Bug 29440: Only log items has been inserted if actually have been so, and remove extra out of place logging of record insertion -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 --- Comment #59 from David Gustafsson <glasklas@gmail.com> --- Created attachment 143375 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=143375&action=edit Bug 29440: Replace removed syspref BiblioAddsAuthorities with AutoLinkBiblios -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #140924|0 |1 is obsolete| | --- Comment #61 from David Gustafsson <glasklas@gmail.com> --- Created attachment 151293 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=151293&action=edit Bug 29440: Refactor and clean up bulkmarcimport.pl To test: 1) Import biblios using ./bulkmarcimport.pl -v -b -file biblios.mrc -insert -update -c=MARC21 -match "control-number,001" -n=1 -l "/tmp/import.log" 2) Verify in /tmp/import.log that one biblio has been inserted 3) Import again without the "-n" option, verify that one biblio has been updated and the rest inserted 4) In the staff interface search the catalog for some string appearing in the imported biblios to verify records have been indexed 5) View/edit some of the biblios the staff interface to very have been properly imported 6) Import authorities using ./bulkmarcimport.pl -v -m=MARCXML -a -file authorities.xml -insert -update -c=MARC21 -keepids="024a" -match "heading-main,100a" -l "/tmp/import.log" -yaml="/tmp/ids.yml" 7) Go to the authorities page and find the imported authority by for example search for a 400 or 100 field value. Verify that the 001 field of the incoming record has been moved to 024a. Also check that the value of 024a appear as a key in /tmp/ids.yml. 8) Edit the authorities.xml file and change the value of field 400a 9) Import authorities again using the same options 10) Verify that the 400a field of the authority in Koha has not been updated (the import was skippped since revision did not change) 11) Edit authorities.xml, replace the value of field 005 in the file with field 005 of the authority in Koha incremented by one 12) Import again with the same options 13) Verify that the 400a field of the authority in Koha now have been updated -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #140926|0 |1 is obsolete| | --- Comment #63 from David Gustafsson <glasklas@gmail.com> --- Created attachment 151295 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=151295&action=edit Bug 29440: Fix QA-tool issues -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #140928|0 |1 is obsolete| | --- Comment #65 from David Gustafsson <glasklas@gmail.com> --- Created attachment 151297 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=151297&action=edit Bug 29440: replace usage of removed GetMarcBiblio -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #140988|0 |1 is obsolete| | --- Comment #67 from David Gustafsson <glasklas@gmail.com> --- Created attachment 151299 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=151299&action=edit Bug 29440: Rename $modify_biblio_marc_options to $mod_bilbio_options This variable actually contains ModBibio options and ModifyBiblioMarc accepts a subset of these -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #143375|0 |1 is obsolete| | --- Comment #69 from David Gustafsson <glasklas@gmail.com> --- Created attachment 151301 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=151301&action=edit Bug 29440: Replace removed syspref BiblioAddsAuthorities with AutoLinkBiblios -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 --- Comment #71 from David Gustafsson <glasklas@gmail.com> --- Created attachment 157798 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=157798&action=edit Bug 29440: Fix AddBiblio/ModBiblio options args -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 --- Comment #73 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to M from comment #72)

I rebased this patch and Bug 25539 in our library and successfully used it to update a lot of biblio and auth records. My feedback is that it generally works, and marc overlay rules are also applied.

There's one biggest issue I've ran into: I've had issues with the -match option with auths. For biblios, I could use `-match Other-control-number,035a` perfectly fine. But for auths, using `-match LC-card-number,010a` just didn't work, so I had to resort to `-match Any,010a`. It works, but not really ideal.

After playing with yaz-client and other tools, I can say this wasn't really indexing issue it seems. I've also ran:

...

xsltproc /etc/koha/zebradb/marc_defs/marc21/authorities/authority-zebra-indexdefs.xsl <(sudo koha-mysql biblioteka <<< "select marcxml from auth_header where ExtractValue( marcxml, '//datafield[@tag=010]/subfield[@code=\"a\"]' ) != '' limit 1\\G" | sed -n 's/marcxml: //;2,$p' ) | grep 2005134806 <z:index name="LC-card-number:w LC-card-number:p">n 2005134806</z:index> <z:index name="Any:w Any:p">n 2005134806</z:index> <z:index name="Any:w Any:p">http://nukat.edu.pl/aut/n 2005134806</z:index>

It seems that the control field is indexed as "LC-card-number", but using it with the scripts always yielded no matches, and with "-update" a warning that insertion of new record was skipped. From my memory of debugging this a few days ago, the $query variable had a value of `LC-card-number="n 2005134806"`.

I also have two more tiny feedback items:

- In the if commented as `#Skip if authority in database is the same or newer than the incoming record`, it skips the record with `next;` with no feedback in logs, neither warn, nor entry in the log like "skipped" similar to "updated"/"inserted". I think it might be a good idea to add some log line here.

- In the final logs it says `n MARC records done in x seconds`. The `n` stands for number of all records that were checked from the file, including those that were skipped. I think it'd be nice if it said something like `n/n MARC records done`, where first `n` would be the number of records actually inserted or updated into the database, while the second `n` would remain the current number. This improvement alone would also partially rectify the problem presented with the first suggestion above.

All in all though it's a very nice changeset, we appreciate that it can load biblio records while respecting MARC overlay rules, it already works for us pretty much as-is, so it'd be nice if it could be merged to 23.11 version already, and maybe tiny details improved later on with time, without blocking this.

The way I see it, this changeset already makes this script MUCH better, fixes broken behavior in it, and doesn't make anything more broken than it already was before, so I definitely vouch for it...

Not sure if I have any authority to review code or make a sign-off, but we used it to import/update thousands of biblio and auth records now, without noticing any issues (I paid special attention to MARC export of biblio before/after).

It sounds like you should sign-off on this then. Do you know how to do it? You can use sandboxes or if you use ktd use the git so command. If you need help, please let us know. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 M <schodkowy.omegi-0r@icloud.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #151293|0 |1 is obsolete| | --- Comment #74 from M <schodkowy.omegi-0r@icloud.com> --- Created attachment 160148 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=160148&action=edit Bug 29440: Refactor and clean up bulkmarcimport.pl -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 M <schodkowy.omegi-0r@icloud.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #151295|0 |1 is obsolete| | --- Comment #76 from M <schodkowy.omegi-0r@icloud.com> --- Created attachment 160150 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=160150&action=edit Bug 29440: Fix QA-tool issues -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 M <schodkowy.omegi-0r@icloud.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #151297|0 |1 is obsolete| | --- Comment #78 from M <schodkowy.omegi-0r@icloud.com> --- Created attachment 160152 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=160152&action=edit Bug 29440: replace usage of removed GetMarcBiblio -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 M <schodkowy.omegi-0r@icloud.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #151299|0 |1 is obsolete| | --- Comment #80 from M <schodkowy.omegi-0r@icloud.com> --- Created attachment 160154 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=160154&action=edit Bug 29440: Rename $modify_biblio_marc_options to $mod_bilbio_options -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 M <schodkowy.omegi-0r@icloud.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #151301|0 |1 is obsolete| | --- Comment #82 from M <schodkowy.omegi-0r@icloud.com> --- Created attachment 160156 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=160156&action=edit Bug 29440: Replace removed syspref BiblioAddsAuthorities with AutoLinkBiblios -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 M <schodkowy.omegi-0r@icloud.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off --- Comment #84 from M <schodkowy.omegi-0r@icloud.com> --- I've signed off on all patches after reviewing them. They improve the situation over what's currently in master, fix inconsistent behavior and don't seem to regress on anything. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 --- Comment #86 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Could you detail the issues you found in some new bugs and link them to this please? It would help me confidently QA this long standing improvement. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #160148|0 |1 is obsolete| | --- Comment #88 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 160477 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=160477&action=edit Bug 29440: Refactor and clean up bulkmarcimport.pl To test: 1) Import biblios using ./bulkmarcimport.pl -v -b -file biblios.mrc -insert -update -c=MARC21 -match "control-number,001" -n=1 -l "/tmp/import.log" 2) Verify in /tmp/import.log that one biblio has been inserted 3) Import again without the "-n" option, verify that one biblio has been updated and the rest inserted 4) In the staff interface search the catalog for some string appearing in the imported biblios to verify records have been indexed 5) View/edit some of the biblios the staff interface to very have been properly imported 6) Import authorities using ./bulkmarcimport.pl -v -m=MARCXML -a -file authorities.xml -insert -update -c=MARC21 -keepids="024a" -match "heading-main,100a" -l "/tmp/import.log" -yaml="/tmp/ids.yml" 7) Go to the authorities page and find the imported authority by for example search for a 400 or 100 field value. Verify that the 001 field of the incoming record has been moved to 024a. Also check that the value of 024a appear as a key in /tmp/ids.yml. 8) Edit the authorities.xml file and change the value of field 400a 9) Import authorities again using the same options 10) Verify that the 400a field of the authority in Koha has not been updated (the import was skippped since revision did not change) 11) Edit authorities.xml, replace the value of field 005 in the file with field 005 of the authority in Koha incremented by one 12) Import again with the same options 13) Verify that the 400a field of the authority in Koha now have been updated Signed-off-by: Michał Kula <148193449+mkibp@users.noreply.github.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #160150|0 |1 is obsolete| | --- Comment #90 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 160479 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=160479&action=edit Bug 29440: Fix QA-tool issues Signed-off-by: Michał Kula <148193449+mkibp@users.noreply.github.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #160152|0 |1 is obsolete| | --- Comment #92 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 160481 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=160481&action=edit Bug 29440: replace usage of removed GetMarcBiblio Signed-off-by: Michał Kula <148193449+mkibp@users.noreply.github.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #160154|0 |1 is obsolete| | --- Comment #94 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 160483 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=160483&action=edit Bug 29440: Rename $modify_biblio_marc_options to $mod_bilbio_options This variable actually contains ModBibio options and ModifyBiblioMarc accepts a subset of these Signed-off-by: Michał Kula <148193449+mkibp@users.noreply.github.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #160156|0 |1 is obsolete| | --- Comment #96 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 160485 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=160485&action=edit Bug 29440: Replace removed syspref BiblioAddsAuthorities with AutoLinkBiblios Signed-off-by: Michał Kula <148193449+mkibp@users.noreply.github.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 --- Comment #98 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 160487 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=160487&action=edit Bug 29440: (QA follow-up) Fix tidy -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 Michaela Sieber <michaela.sieber@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |michaela.sieber@kit.edu -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to master Version(s)| |24.05.00 released in| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m --- Comment #102 from Fridolin Somers <fridolin.somers@biblibre.com> --- Enhancement not pushed to 23.11.x -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Component|Tools |Command-line Utilities CC| |robin@catalyst.net.nz -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 Jan Kissig <bibliothek@th-wildau.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |37020 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37020 [Bug 37020] bulkmarcimport gets killed after update to 24.05. when inserting large files -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC|martin.renvoize@ptfs-europe | |.com | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29440 --- Comment #106 from Michał <schodkowy.omegi-0r@icloud.com> --- True, I still have it in my backlog to poke around that script sometime, but an opportunity could come within 2-3 months if we'll be refactoring our import scripts... (I also had the idea in my head to add some second simpler script to manipulate/swap just a single MARC record with external tooling/scripts) -- You are receiving this mail because: You are watching all bug changes.