[Bug 33576] New: Records are not indexed when imported if using Elasticsearch
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33576 Bug ID: 33576 Summary: Records are not indexed when imported if using Elasticsearch Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: major Priority: P5 - low Component: Tools Assignee: koha-bugs@lists.koha-community.org Reporter: nick@bywatersolutions.com QA Contact: testopia@bugs.koha-community.org The importing is done in a transaction. We call the indexer form inside the transaction. This means that when rabbit sends the task, it may not be found in the DB yet. We need to ensure we index after the transaction completes -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33576 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33576 --- Comment #1 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 149971 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=149971&action=edit Bug 33576: Index records after import transaction is committed This patch simply moves our indexing call after the transaction is committed so that the job will exist in the DB when called. To test: 1 - Have Koha using Elasticsearch 2 - Stage and import a file of records 3 - View the job in Admin->Manage jobs 4 - Note it is not finished 5 - Check log: /var/log/koha/kohadev/es-indexer-output.log 6 - Note: [WARN] No job found for id=### 7 - Apply patch 8 - Stage and import 9 - Note no error in log 10 - Note successful completion of indexing job -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33576 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |nick@bywatersolutions.com |ity.org | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33576 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |27421 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 [Bug 27421] Porting tools/stage-marc-import.pl to BackgroundJob -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33576 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |m.de.rooy@rijksmuseum.nl -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33576 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david@davidnind.com --- Comment #2 from David Nind <david@davidnind.com> --- I'm successfully able to stage and import records and items (tried imports with and without items). I'm not seeing the warning in the log. However, I do have this error in /var/log/koha/kohadev/es-indexer-output.log - but this is there before doing any staging and importing: Cannot connect to the message broker, the jobs will be processed anyway (Failed to connect: Error connecting to localhost:61613: Cannot assign requested address at /usr/share/perl5/Net/Stomp.pm line 27.; giving up at /usr/share/perl5/Net/Stomp.pm line 27. ) at /kohadevbox/koha/misc/workers/es_indexer_daemon.pl line 84. I've logged this as an issue for KTD https://gitlab.com/koha-community/koha-testing-docker/-/issues/373 Maybe this is why I can't replicate the issue for this bug? Otherwise, is there anything else I need to be doing? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33576 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|--- |Small patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33576 --- Comment #3 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 150147 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=150147&action=edit Bug 33576: (QA follow-up) Polish comment, typo No test plan. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33576 --- Comment #4 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Feels like a race condition. So it depends who was faster: rabbitmq reading a new ticket or BatchCommitRecords doing the txn_commit. Agree with this patch. Happy to QA it. Do not have Elastic at hand for testing though. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33576 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |m.de.rooy@rijksmuseum.nl |y.org | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33576 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33576 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #149971|0 |1 is obsolete| | Attachment #150147|0 |1 is obsolete| | --- Comment #5 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 150591 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=150591&action=edit Bug 33576: Index records after import transaction is committed This patch simply moves our indexing call after the transaction is committed so that the job will exist in the DB when called. To test: 1 - Have Koha using Elasticsearch 2 - Stage and import a file of records 3 - View the job in Admin->Manage jobs 4 - Note it is not finished 5 - Check log: /var/log/koha/kohadev/es-indexer-output.log 6 - Note: [WARN] No job found for id=### 7 - Apply patch 8 - Stage and import 9 - Note no error in log 10 - Note successful completion of indexing job Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33576 --- Comment #6 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 150592 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=150592&action=edit Bug 33576: (QA follow-up) Polish comment, typo No test plan. Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33576 Bug 33576 depends on bug 27421, which changed state. Bug 27421 Summary: Porting tools/stage-marc-import.pl to BackgroundJob https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27421 What |Removed |Added ---------------------------------------------------------------------------- Status|Needs documenting |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33576 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33576 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #150591|0 |1 is obsolete| | --- Comment #7 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 150863 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=150863&action=edit Bug 33576: Index records after import transaction is committed This patch simply moves our indexing call after the transaction is committed so that the job will exist in the DB when called. To test: 1 - Have Koha using Elasticsearch 2 - Stage and import a file of records 3 - View the job in Admin->Manage jobs 4 - Note it is not finished 5 - Check log: /var/log/koha/kohadev/es-indexer-output.log 6 - Note: [WARN] No job found for id=### 7 - Apply patch 8 - Stage and import 9 - Note no error in log 10 - Note successful completion of indexing job Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33576 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #150592|0 |1 is obsolete| | --- Comment #8 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 150864 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=150864&action=edit Bug 33576: (QA follow-up) Polish comment, typo No test plan. Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33576 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to master Version(s)| |23.05.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33576 --- Comment #9 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Pushed to master for 23.05. Nice work everyone, thanks! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33576 Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|23.05.00 |23.05.00,22.11.06 released in| | Status|Pushed to master |Pushed to stable -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33576 --- Comment #10 from Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com> --- Nice work everyone! Pushed to stable for 22.11.x -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33576 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED CC| |lucas@bywatersolutions.com Status|Pushed to stable |RESOLVED --- Comment #11 from Lucas Gass <lucas@bywatersolutions.com> --- Missing dependencies for 22.05.x, no backport. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33576 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |aleisha@catalyst.net.nz --- Comment #12 from Aleisha Amohia <aleisha@catalyst.net.nz> --- *** Bug 35156 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org