[Bug 35466] New: bulkmarcimport needs a parameter to skip indexing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35466 Bug ID: 35466 Summary: bulkmarcimport needs a parameter to skip indexing Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Command-line Utilities Assignee: koha-bugs@lists.koha-community.org Reporter: katrin.fischer@bsz-bw.de QA Contact: testopia@bugs.koha-community.org CC: jonathan.druart+koha@gmail.com, michaela.sieber@kit.edu, robin@catalyst.net.nz When using bulkmarcimport for migrations it does make sense to do a full reindex after finishing the import. At the moment the import will always create entries in background_jobs for Elasticsearch and in zebra_queue for Zebra. We should have a switch to turn this off if indexing is not required to increase performance and avoid unnecessary load. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35466 --- Comment #1 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- We need to adjust the calls to: AddBiblio ModBiblio ModBiblioMarc AddItemBatchFromMarc And add the skip_record_index parameter. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35466 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nick@bywatersolutions.com --- Comment #2 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- *** Bug 37565 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35466 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35466 --- Comment #3 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 170059 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=170059&action=edit Bug 35466: Add skip_indexing parameter to bulkmarciport.pl This patch adds a new optoin to skip indexing to bulkmarcimport. It also fixes a bug where authorities were being indexed multiple times during import. To test: 1 - Apply patch 2 - Download the sample files on this bug 3 - Perform asearch engine search that should retrieve all records( 'a' in Zebra, '*' in ES) 4 - Note the total (435 in KTD) 5 - perl misc/migration_tools/bulkmarcimport.pl -b -v --file=bug_35466_b.mrc 6 - Search again, note increaed by 100 7 - perl misc/migration_tools/bulkmarcimport.pl -b -v --file=bug_35466_b.mrc --skip_indexing 8 - Search again, no increase 9 - perl misc/search_tools/rebuild_elasticsearch.pl -v 10 - Search again, increase, records were added but not initially indexed 11 - Browse to authorities and search as for biblios 12 - perl misc/migration_tools/bulkmarcimport.pl -a -v --file=bug_35466_a.mrc 13 - Search again, note increase 14 - perl misc/migration_tools/bulkmarcimport.pl -a -v --file=bug_35466_a.mrc --skip_indexing 15 - Search again, no increase 16 - perl misc/search_tools/rebuild_elasticsearch.pl -v 17 - Search again, increase, records were added but not initially indexed 18 - Sign off! Hi5! -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35466 --- Comment #4 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 170060 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=170060&action=edit Test auth file -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35466 --- Comment #5 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 170061 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=170061&action=edit Test biblios file -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35466 Phil Ringnalda <phil@chetcolibrary.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |nick@bywatersolutions.com |ity.org | CC| |phil@chetcolibrary.org -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35466 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=37478 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35466 --- Comment #6 from Phil Ringnalda <phil@chetcolibrary.org> --- Test plan fails for me at step 13 while using ElasticSearch, there's no increase. Were authorities really being indexed multiple times? The way the indexing call is $indexer->update_index( \@search_engine_record_ids, \@search_engine_records ); and those params are only set for biblios, it looks to me from a quick skim of update_index() that for authorities we don't actually index if we don't index in ModAuthority/AddAuthority, because passing empty @record_ids and @records just indexes nothing. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35466 Phil Ringnalda <phil@chetcolibrary.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35466 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35466 --- Comment #7 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 170631 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=170631&action=edit Bug 35466: (follow-up) Go to next if error, add to indexing call if not Before the previous patch we were indexing with every AddAuthority/ModAuthority call and I assumed we were also indexing during the commity, however, it appears we were not. This patch ensures we push the record and ids into the arrays for indexing during commit. Additionally I add a skip to next record on error, to match biblio behaviour. I also correct a log referring to biblios during authority importing -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35466 --- Comment #8 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- (In reply to Phil Ringnalda from comment #6)
Test plan fails for me at step 13 while using ElasticSearch, there's no increase.
Were authorities really being indexed multiple times? The way the indexing call is $indexer->update_index( \@search_engine_record_ids, \@search_engine_records ); and those params are only set for biblios, it looks to me from a quick skim of update_index() that for authorities we don't actually index if we don't index in ModAuthority/AddAuthority, because passing empty @record_ids and @records just indexes nothing.
Ah, I assumed the code was doing the right thing as well as the wrong thing, I was wrong ;-) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35466 Phil Ringnalda <phil@chetcolibrary.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35466 Phil Ringnalda <phil@chetcolibrary.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #170059|0 |1 is obsolete| | --- Comment #9 from Phil Ringnalda <phil@chetcolibrary.org> --- Created attachment 170641 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=170641&action=edit Bug 35466: Add skip_indexing parameter to bulkmarciport.pl This patch adds a new optoin to skip indexing to bulkmarcimport. It also fixes a bug where authorities were being indexed multiple times during import. To test: 1 - Apply patch 2 - Download the sample files on this bug 3 - Perform asearch engine search that should retrieve all records( 'a' in Zebra, '*' in ES) 4 - Note the total (435 in KTD) 5 - perl misc/migration_tools/bulkmarcimport.pl -b -v --file=bug_35466_b.mrc 6 - Search again, note increaed by 100 7 - perl misc/migration_tools/bulkmarcimport.pl -b -v --file=bug_35466_b.mrc --skip_indexing 8 - Search again, no increase 9 - perl misc/search_tools/rebuild_elasticsearch.pl -v 10 - Search again, increase, records were added but not initially indexed 11 - Browse to authorities and search as for biblios 12 - perl misc/migration_tools/bulkmarcimport.pl -a -v --file=bug_35466_a.mrc 13 - Search again, note increase 14 - perl misc/migration_tools/bulkmarcimport.pl -a -v --file=bug_35466_a.mrc --skip_indexing 15 - Search again, no increase 16 - perl misc/search_tools/rebuild_elasticsearch.pl -v 17 - Search again, increase, records were added but not initially indexed 18 - Sign off! Hi5! Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35466 Phil Ringnalda <phil@chetcolibrary.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #170631|0 |1 is obsolete| | --- Comment #10 from Phil Ringnalda <phil@chetcolibrary.org> --- Created attachment 170642 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=170642&action=edit Bug 35466: (follow-up) Go to next if error, add to indexing call if not Before the previous patch we were indexing with every AddAuthority/ModAuthority call and I assumed we were also indexing during the commity, however, it appears we were not. This patch ensures we push the record and ids into the arrays for indexing during commit. Additionally I add a skip to next record on error, to match biblio behaviour. I also correct a log referring to biblios during authority importing Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35466 Thomas Klausner <domm@plix.at> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |domm@plix.at |y.org | CC| |domm@plix.at -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35466 --- Comment #11 from Thomas Klausner <domm@plix.at> --- Created attachment 172997 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=172997&action=edit Bug 35466: (follow-up) document skip_indexing parameter to bulkmarciport.pl Add documentation for the new -si, --skip_indexing parameter -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35466 Thomas Klausner <domm@plix.at> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #170641|0 |1 is obsolete| | --- Comment #12 from Thomas Klausner <domm@plix.at> --- Created attachment 172998 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=172998&action=edit Bug 35466: Add skip_indexing parameter to bulkmarciport.pl This patch adds a new optoin to skip indexing to bulkmarcimport. It also fixes a bug where authorities were being indexed multiple times during import. To test: 1 - Apply patch 2 - Download the sample files on this bug 3 - Perform asearch engine search that should retrieve all records( 'a' in Zebra, '*' in ES) 4 - Note the total (435 in KTD) 5 - perl misc/migration_tools/bulkmarcimport.pl -b -v --file=bug_35466_b.mrc 6 - Search again, note increaed by 100 7 - perl misc/migration_tools/bulkmarcimport.pl -b -v --file=bug_35466_b.mrc --skip_indexing 8 - Search again, no increase 9 - perl misc/search_tools/rebuild_elasticsearch.pl -v 10 - Search again, increase, records were added but not initially indexed 11 - Browse to authorities and search as for biblios 12 - perl misc/migration_tools/bulkmarcimport.pl -a -v --file=bug_35466_a.mrc 13 - Search again, note increase 14 - perl misc/migration_tools/bulkmarcimport.pl -a -v --file=bug_35466_a.mrc --skip_indexing 15 - Search again, no increase 16 - perl misc/search_tools/rebuild_elasticsearch.pl -v 17 - Search again, increase, records were added but not initially indexed 18 - Sign off! Hi5! Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org> Signed-off-by: Thomas Klausner <domm@plix.at> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35466 Thomas Klausner <domm@plix.at> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #170642|0 |1 is obsolete| | --- Comment #13 from Thomas Klausner <domm@plix.at> --- Created attachment 172999 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=172999&action=edit Bug 35466: (follow-up) Go to next if error, add to indexing call if not Before the previous patch we were indexing with every AddAuthority/ModAuthority call and I assumed we were also indexing during the commity, however, it appears we were not. This patch ensures we push the record and ids into the arrays for indexing during commit. Additionally I add a skip to next record on error, to match biblio behaviour. I also correct a log referring to biblios during authority importing Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org> Signed-off-by: Thomas Klausner <domm@plix.at> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35466 Thomas Klausner <domm@plix.at> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #172997|0 |1 is obsolete| | --- Comment #14 from Thomas Klausner <domm@plix.at> --- Created attachment 173000 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=173000&action=edit Bug 35466: (follow-up) document skip_indexing parameter to bulkmarciport.pl Add documentation for the new -si, --skip_indexing parameter Signed-off-by: Thomas Klausner <domm@plix.at> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35466 Thomas Klausner <domm@plix.at> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA --- Comment #15 from Thomas Klausner <domm@plix.at> --- Interestingly enough we needed exactly this feature today during a migration, so I QA'ed it and it works as advertised! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35466 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to main Version(s)| |24.11.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35466 --- Comment #16 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Pushed for 24.11! Well done everyone, thank you! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35466 --- Comment #17 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Fixed typo in bulkmarciport.pl in commit message on push. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35466 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |release-notes-needed -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35466 Lucas Gass (lukeg) <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to main |Pushed to stable Version(s)|24.11.00 |24.11.00,24.05.06 released in| | CC| |lucas@bywatersolutions.com --- Comment #18 from Lucas Gass (lukeg) <lucas@bywatersolutions.com> --- Backported to 24.05.x for upcoming 24.05.06 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35466 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the| |This patch adds a new release notes| |option to skip indexing to | |bulkmarcimport: | |`--skip_indexing` | |It also | |fixes a bug where | |authorities were being | |indexed multiple times | |during import. Keywords|release-notes-needed | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35466 Lucas Gass (lukeg) <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to stable |Pushed to oldstable -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35466 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m --- Comment #19 from Fridolin Somers <fridolin.somers@biblibre.com> --- Does not apply easy on 23.11.x Please provided a specific patch if you can -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35466 Wainui Witika-Park <wainuiwitikapark@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |wainuiwitikapark@catalyst.n | |et.nz Status|Pushed to oldstable |Needs documenting --- Comment #20 from Wainui Witika-Park <wainuiwitikapark@catalyst.net.nz> --- Not backporting to 22.11 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35466 --- Comment #21 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- We used the parameter in latest 24.11, but when I checked I still saw the entries in the background_jobs table. Can someone confirm? -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org