[Bug 35438] New: Importing records can create too large transactions
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35438 Bug ID: 35438 Summary: Importing records can create too large transactions Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Tools Assignee: koha-bugs@lists.koha-community.org Reporter: nick@bywatersolutions.com QA Contact: testopia@bugs.koha-community.org When importing a staged file we commit only every 50 records For an authority import we are also merging, which can affect many more biblios, and these all end up in the transaction. This can cause tables locks and issues across Koha Why do we not commit each record as we go? -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35438 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au, | |jonathan.druart+koha@gmail. | |com, | |kyle@bywatersolutions.com, | |martin.renvoize@ptfs-europe | |.com, tomascohen@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35438 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=35439 -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35438 --- Comment #1 from David Cook <dcook@prosentient.com.au> --- (In reply to Nick Clemens from comment #0)
When importing a staged file we commit only every 50 records
For an authority import we are also merging, which can affect many more biblios, and these all end up in the transaction.
This can cause tables locks and issues across Koha
Why do we not commit each record as we go?
Historically, it was probably for speed reasons, but since we're using background jobs I think committing each record as we go should be fine in terms of performance. Back in the day, we didn't use transactions either, so they wouldn't have been an issue back then. I work on another system which is quite busy, and I've had to commit more frequently to avoid deadlocks and other transaction related issues. It is much slower but it's much more reliable. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35438 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35438 --- Comment #2 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 159413 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=159413&action=edit Bug 35438: Transact each record import separately When importing a staged file we commit every 50 records For an authority import we are also merging, which can affect many more biblios, and these all end up in the transaction. This can cause tables locks and issues across Koha Test Plan: 1) Apply this patch 2) prove t/db_dependent/ImportBatch.t -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35438 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |kyle@bywatersolutions.com |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35438 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35438 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #159413|0 |1 is obsolete| | --- Comment #3 from David Nind <david@davidnind.com> --- Created attachment 159415 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=159415&action=edit Bug 35438: Transact each record import separately When importing a staged file we commit every 50 records For an authority import we are also merging, which can affect many more biblios, and these all end up in the transaction. This can cause tables locks and issues across Koha Test Plan: 1) Apply this patch 2) prove t/db_dependent/ImportBatch.t Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35438 --- Comment #4 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- The idea of surrounding the whole job with a transaction was to be able to "retry" it in case it fails. If we process only a subset it won't be possible (unless parsing the "report"). -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35438 --- Comment #5 from Kyle M Hall <kyle@bywatersolutions.com> --- (In reply to Jonathan Druart from comment #4)
The idea of surrounding the whole job with a transaction was to be able to "retry" it in case it fails. If we process only a subset it won't be possible (unless parsing the "report").
I think it was a great theory and and bad practice. Especially since we are transacting 50 at a time and not entire batches ( perish the thought! ). -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35438 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #159415|0 |1 is obsolete| | --- Comment #6 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 159442 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=159442&action=edit Bug 35438: Transact each record import separately When importing a staged file we commit every 50 records For an authority import we are also merging, which can affect many more biblios, and these all end up in the transaction. This can cause tables locks and issues across Koha Test Plan: 1) Apply this patch 2) prove t/db_dependent/ImportBatch.t Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35438 --- Comment #7 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 159443 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=159443&action=edit Bug 35438: Remove skip_intermediate_commit parameter It no longer does anything. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35438 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |m.de.rooy@rijksmuseum.nl |y.org | Status|Signed Off |BLOCKED CC| |m.de.rooy@rijksmuseum.nl --- Comment #8 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Looking here -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35438 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|BLOCKED |Passed QA Patch complexity|--- |Small patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35438 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #159442|0 |1 is obsolete| | --- Comment #9 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 160917 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=160917&action=edit Bug 35438: Transact each record import separately When importing a staged file we commit every 50 records For an authority import we are also merging, which can affect many more biblios, and these all end up in the transaction. This can cause tables locks and issues across Koha Test Plan: 1) Apply this patch 2) prove t/db_dependent/ImportBatch.t Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35438 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #159443|0 |1 is obsolete| | --- Comment #10 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 160918 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=160918&action=edit Bug 35438: Remove skip_intermediate_commit parameter It no longer does anything. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35438 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |24.05.00 released in| | Status|Passed QA |Pushed to master -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35438 --- Comment #11 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Pushed for 24.05! Well done everyone, thank you! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35438 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m Version(s)|24.05.00 |24.05.00,23.11.02 released in| | Status|Pushed to master |Pushed to stable --- Comment #12 from Fridolin Somers <fridolin.somers@biblibre.com> --- Pushed to 23.11.x for 23.11.02 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35438 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|24.05.00,23.11.02 |24.05.00,23.11.02,23.05.09 released in| | Status|Pushed to stable |Pushed to oldstable CC| |lucas@bywatersolutions.com --- Comment #13 from Lucas Gass <lucas@bywatersolutions.com> --- Backported to 23.05.x for upcoming 23.05.09 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35438 Michelle Spinney <mspinney@clamsnet.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mspinney@clamsnet.org -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35438 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC|martin.renvoize@ptfs-europe | |.com | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35438 Wainui Witika-Park <wainuiwitikapark@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|24.05.00,23.11.02,23.05.09 |24.05.00,23.11.02,23.05.09, released in| |22.11.28 CC| |wainuiwitikapark@catalyst.n | |et.nz Resolution|--- |FIXED Status|Pushed to oldstable |RESOLVED --- Comment #14 from Wainui Witika-Park <wainuiwitikapark@catalyst.net.nz> --- Applied to 22.11 for 22.11.28 -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org