https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37564 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=37478 -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37564 Martin Renvoize (ashimema) <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |martin.renvoize@ptfs-europe |ity.org |.com CC| |martin.renvoize@ptfs-europe | |.com -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37564 Martin Renvoize (ashimema) <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bernard.scaife@ptfs-europe. | |com -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37564 Martin Renvoize (ashimema) <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|normal |major -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37564 --- Comment #4 from Magnus Enger <magnus@libriotech.no> --- I have confirmed the problem with bulkmarcimport.pl and reported it as a separate bug: Bug 38962 -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37564 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david@davidnind.com --- Comment #6 from David Nind <david@davidnind.com> --- I had a go at testing the patch for main (thanks Magnus for the test plan!). However, it still seems to be updating the holds queue. Testing notes (using KTD): 1. Enable the real times hold queue: RealTimeHoldsQueue = Enable 2. Delete any background jobs: DELETE FROM background_jobs; 3. Check the number of holds queue jobs in the database: SELECT COUNT(*) FROM background_jobs WHERE type = update_holds_queue_for_biblios'; 4. Export a record with items as XML from Koha. 5. Delete the exported record from Koha 6. Import the record, using bulkmarcimport.pl: misc/migration_tools/bulkmarcimport.pl -b -v -m=MARCXML --file my_record.marcxml 7. Check the jobs in the database, there should be one per item imported. 8. Delete the imported record 9. Delete all background jobs (see step 2). 10. Apply the patch 11. Import the record again (see step 6) 12. Check the number of holds queue jobs in the database again - should be 0. Here is the database entries after the patch is applied (probably hard to read): | id | status | progress | size | borrowernumber | type | queue | data | context | enqueued_on | started_on | ended_on | +----+----------+----------+------+----------------+--------------------------------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------+---------------------+---------------------+ | 3 | finished | 1 | 1 | NULL | update_holds_queue_for_biblios | default | {"messages":[{"type":"success","code":"holds_queue_updated","biblio_id":439}],"report":{"total_biblios":1,"total_success":1},"biblio_ids":[439]} | {"cardnumber":null,"number":null,"interface":"commandline","firstname":"CLI","desk_name":null,"desk_id":null,"surname":"CLI","register_name":null,"branch":null,"register_id":null,"branchname":null,"shibboleth":null,"emailaddress":null,"id":null,"flags":null} | 2025-01-24 18:51:49 | 2025-01-24 18:51:50 | 2025-01-24 18:51:50 | +----+----------+----------+------+----------------+--------------------------------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------+---------------------+---------------------+ Other notes: 1. I reindexed after deleting the record, probably diidn't need to: koha-rebuild-zebra -d -f -v kohadev 2. I used a different record after applying the patch, as I had an error when trying to delete the imported record. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37564 --- Comment #7 from Magnus Enger <magnus@libriotech.no> --- I see the same thing as David, there are still backgroundjobs with type = update_holds_queue_for_biblios after applying the patch. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37564 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37564 --- Comment #9 from David Nind <david@davidnind.com> --- I'm getting a syntax error after applying the patches: misc/migration_tools/bulkmarcimport.pl -b -v -m=MARCXML --file bib-11.marcxml syntax error at misc/migration_tools/bulkmarcimport.pl line 143, near "$skip_indexing skip_holds_queue" BEGIN not safe after errors--compilation aborted at misc/migration_tools/bulkmarcimport.pl line 152. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37564 Magnus Enger <magnus@libriotech.no> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA --- Comment #11 from Magnus Enger <magnus@libriotech.no> --- Sorry, this is still not working for me. I import a record: $ perl /usr/share/koha/bin/migration_tools/bulkmarcimport.pl -b -v -m MARCXML --file lanark.marcxml Characteristic MARC flavour: MARC21 Use of uninitialized value in concatenation (.) or string at /usr/share/perl5/MARC/File/XML.pm line 399, <GEN3> chunk 3. . 1 MARC records done in 0.28852391242981 seconds And then there are holds jobs: MariaDB [koha_kohadev]> select enqueued_on, type, queue, status from background_jobs; +---------------------+--------------------------------+---------------+----------+ | enqueued_on | type | queue | status | +---------------------+--------------------------------+---------------+----------+ | 2025-01-29 07:52:33 | update_elastic_index | elastic_index | finished | | 2025-01-29 07:52:33 | update_holds_queue_for_biblios | default | finished | | 2025-01-29 07:52:33 | update_elastic_index | elastic_index | finished | | 2025-01-29 07:52:33 | update_holds_queue_for_biblios | default | finished | | 2025-01-29 07:52:33 | update_elastic_index | elastic_index | finished | | 2025-01-29 07:52:33 | update_holds_queue_for_biblios | default | finished | | 2025-01-29 07:52:33 | update_elastic_index | elastic_index | finished | | 2025-01-29 07:52:33 | update_holds_queue_for_biblios | default | finished | +---------------------+--------------------------------+---------------+----------+ 8 rows in set (0.000 sec) -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37564 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #176186|0 |1 is obsolete| | Attachment #177199|0 |1 is obsolete| | Attachment #177210|0 |1 is obsolete| | --- Comment #12 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 188704 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=188704&action=edit Bug 37564: Prevent bulkmarcimport from generating holds queue jobs We shouldn't need to update the holds queue for such command line imports when only adding/updating a biblio. When we add a biblio and add items, we do not need to rebuild the holds queue as there can't be holds on a new record If the record was matched and then updated or ignored we will need to rebuild the queue as we add items. TO DO: - Right now we rebuild per item added, really we should probably skip the entire build until the end and handle here. - We should also delay record index builds until all items have been added - currently we will reindex once per item Testing notes (using KTD): 1. Enable the real times hold queue: RealTimeHoldsQueue = Enable 2. Delete any background jobs: DELETE FROM background_jobs; 3. Check the number of holds queue jobs in the database: SELECT COUNT(*) FROM background_jobs WHERE type = update_holds_queue_for_biblios'; 4. Export a record with items as XML from Koha. 5. Delete the exported record from Koha 6. Import the record, using bulkmarcimport.pl: misc/migration_tools/bulkmarcimport.pl -b -v -m=MARCXML --file my_record.marcxml 7. Check the jobs in the database, there should be one per item imported. 8. Delete the imported record 9. Delete all background jobs (see step 2). 10. Apply the patch 11. Import the record again (see step 6) 12. Check the number of holds queue jobs in the database again - should be 0. 13. Import biblio and use matchin option to overlay previous import 14. Check the number of holds queue jobs in the database again - should be one per item. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37564 Jan Kissig <bibliothek@th-wildau.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bibliothek@th-wildau.de See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=41681 -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37564 --- Comment #15 from David Nind <david@davidnind.com> --- (In reply to Katrin Fischer from comment #14)

Hi David, I believe you need to use the bulkmarcimport.pl script here as the change is for this type of import and (not yet) for staged import.

Thanks Katrin! -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37564 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37564 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #188704|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37564 --- Comment #18 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 200729 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200729&action=edit Bug 37564: Prevent bulkmarcimport from generating holds queue jobs When importing records from the command line we do not always need to rebuild the holds queue. Adding a brand new biblio and its items cannot affect the holds queue, as there can be no holds against a record that did not previously exist, so the rebuild can be skipped entirely. When an existing record is matched and updated, however, the queue does need rebuilding as we (re)add the items, because holds may already exist against that record. To support this, AddItemBatchFromMarc now accepts an options hashref and passes a skip_holds_queue flag down to Koha::Item->store. bulkmarcimport sets the flag per record: skipping the rebuild on insert and allowing it on update. The biblio level options also skip the holds queue, as any required rebuild is driven by the item additions. Test plan (using KTD): 1. Enable the real time holds queue: RealTimeHoldsQueue = Enable 2. Delete any existing background jobs: DELETE FROM background_jobs; 3. Count the holds queue jobs (should be 0): SELECT COUNT(*) FROM background_jobs WHERE type = 'update_holds_queue_for_biblios'; 4. Export a record that has at least one item as MARCXML from Koha. 5. Delete the exported record from Koha. 6. Import the record with bulkmarcimport.pl (insert is the default): misc/migration_tools/bulkmarcimport.pl -b -v -m=MARCXML \ --file my_record.marcxml 7. Re-run the query from step 3: there is one job per item imported. 8. Delete the imported record again. 9. Clear the background jobs again (see step 2). 10. Apply this patch. 11. Import the record again (see step 6). 12. Re-run the query from step 3: the count is now 0, because the record is new and cannot have holds. 13. Clear the background jobs again, then import the record once more, this time matching and *updating* the existing record so its items are re-added. The --update flag is required: without it a matched record is skipped and no items are added (this is why no jobs appeared in earlier testing). Make sure the record from step 11 has finished indexing so it can be matched: misc/migration_tools/bulkmarcimport.pl -b -v -m=MARCXML --update \ --match=isbn,020a --file my_record.marcxml 14. Re-run the query from step 3: there is one job per item, because the items were added to an existing record that may have holds. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37564 --- Comment #20 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 200731 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200731&action=edit Bug 37564: Rebuild holds queue and reindex once per batch, not per item AddItemBatchFromMarc previously let Koha::Item->store rebuild the holds queue and reindex the biblio as each item was stored, so importing a record with N items triggered N holds queue jobs and N reindexes for a single biblio. Defer both operations: each item is now stored with skip_record_index and skip_holds_queue set, and the biblio is reindexed and the holds queue rebuilt a single time once the whole batch has been added. The caller can still suppress either step via the $options hashref, and bulkmarcimport now passes skip_record_index through in line with the biblio level options (so --skip_indexing and the Elasticsearch batch indexer also cover the item additions). This resolves the two TODOs noted on the previous patch. As a result, updating an existing record now generates a single holds queue job per record rather than one per item. Test plan: 1. Enable RealTimeHoldsQueue and clear background_jobs. 2. Import a record with several items, matching and *updating* an existing record (see the holds queue patch test plan, step 13). 3. Confirm only a single update_holds_queue_for_biblios job is created for the record, regardless of the number of items. 4. Confirm the imported items are searchable (the reindex still happens, once). 5. Repeat with --skip_indexing and confirm no reindex is triggered. 6. Repeat on an Elasticsearch system and confirm items are indexed via the batch indexer. 7. prove t/db_dependent/Labels/t_Batch.t -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37564 --- Comment #21 from Jan Kissig <bibliothek@th-wildau.de> --- (In reply to Martin Renvoize (ashimema) from comment #17)

Created attachment 200728 [details] [review] Bug 37564: Fix double increment of record_number in bulkmarcimport.pl

$record_number was incremented twice for every record processed: once unconditionally at the top of the RECORD loop and again inside the "if ($record)" block that handles a successfully fetched record. The duplicate increment was introduced by the refactoring in bug 29440.

This threw off everything keyed on the counter: the progress output, the per-commitnum transaction batching and the --number limit (which could be overshot when an odd limit never matched the doubled count).

Drop the unconditional increment so each record is counted exactly once, inside the block that confirms a record was actually read.

Test plan: 1. Run a bulkmarcimport with --number set to an odd value, e.g. misc/migration_tools/bulkmarcimport.pl -b -m=MARCXML --number=3 \ --file my_records.marcxml 2. Without this patch, note the reported count and that --number is not respected as expected. 3. Apply the patch. 4. Repeat the import and confirm exactly --number records are processed and the final "N MARC records done" count is correct.

I hope this does not interferere with 41681 as record_number is fixed there -- You are receiving this mail because: You are watching all bug changes.