[Bug 30824] New: Improve performance of BatchCommitItems
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30824 Bug ID: 30824 Summary: Improve performance of BatchCommitItems Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Architecture, internals, and plumbing Assignee: koha-bugs@lists.koha-community.org Reporter: nick@bywatersolutions.com QA Contact: testopia@bugs.koha-community.org -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30824 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |30813 Status|NEW |Needs Signoff Patch complexity|--- |Small patch Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30813 [Bug 30813] Refactor TransformMarcToKoha to remove TransformMarcToKohaOneField -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30824 --- Comment #1 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 135247 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=135247&action=edit Bug 30824: Improve performance of BatchCommitItems This makes two simple changes: - Limit TransformMarcToKoha to the fields we need - Pass forward the biblioitemnumber when adding items to a new biblionumber Profiling with NYTProf I saved ~8-9 seconds importing around 400 bibs/1000 items Reducing calls in item store to use a passed biblionumber was the largest gain. To test: 1 - Import some records and items 2 - Verify values etc., revert 3 - Apply patch 4 - Import again 5 - Verify values etc. same as before -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30824 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |nick@bywatersolutions.com |ity.org | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30824 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart+koha@gmail. | |com See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=30822 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30824 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |martin.renvoize@ptfs-europe | |.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30824 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #135247|0 |1 is obsolete| | --- Comment #2 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 135307 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=135307&action=edit Bug 30824: Improve performance of BatchCommitItems This makes two simple changes: - Limit TransformMarcToKoha to the fields we need - Pass forward the biblioitemnumber when adding items to a new biblionumber Profiling with NYTProf I saved ~8-9 seconds importing around 400 bibs/1000 items Reducing calls in item store to use a passed biblionumber was the largest gain. To test: 1 - Import some records and items 2 - Verify values etc., revert 3 - Apply patch 4 - Import again 5 - Verify values etc. same as before Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30824 --- Comment #3 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 135308 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=135308&action=edit Bug 30824: (follow-up) POD -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30824 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off --- Comment #4 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Strong performance improvement, signing off. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30824 Joonas Kylmälä <joonas.kylmala@iki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |joonas.kylmala@iki.fi |y.org | CC| |joonas.kylmala@iki.fi -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30824 Joonas Kylmälä <joonas.kylmala@iki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30824 Joonas Kylmälä <joonas.kylmala@iki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #135307|0 |1 is obsolete| | Attachment #135308|0 |1 is obsolete| | --- Comment #5 from Joonas Kylmälä <joonas.kylmala@iki.fi> --- Created attachment 135369 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=135369&action=edit Bug 30824: Improve performance of BatchCommitItems This makes two simple changes: - Limit TransformMarcToKoha to the fields we need - Pass forward the biblioitemnumber when adding items to a new biblionumber Profiling with NYTProf I saved ~8-9 seconds importing around 400 bibs/1000 items Reducing calls in item store to use a passed biblionumber was the largest gain. To test: 1 - Import some records and items 2 - Verify values etc., revert 3 - Apply patch 4 - Import again 5 - Verify values etc. same as before Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30824 --- Comment #6 from Joonas Kylmälä <joonas.kylmala@iki.fi> --- Created attachment 135370 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=135370&action=edit Bug 30824: (follow-up) POD Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30824 --- Comment #7 from Joonas Kylmälä <joonas.kylmala@iki.fi> --- Makes the code harder to understand but I'll allow it due to the perf improvement. Passing QA. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30824 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |tomascohen@gmail.com --- Comment #8 from Tomás Cohen Arazi <tomascohen@gmail.com> --- How does the dependency work, Nick? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30824 --- Comment #9 from Nick Clemens <nick@bywatersolutions.com> --- (In reply to Tomás Cohen Arazi from comment #8)
How does the dependency work, Nick?
It comes from the updated call to TransformMarcToKoha - we could simply use 'items' in the limit_table param to remove it -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30824 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |BLOCKED --- Comment #10 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Blocked by tree. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30824 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|BLOCKED |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30824 --- Comment #11 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Blocking patch has been pushed now, setting back to PQA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30824 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to master Version(s)| |22.11.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30824 --- Comment #12 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Pushed to master for 22.11. Nice work everyone, thanks! -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org