[Bug 37682] New: Improve speed of koha-preferences CLI tool (by lazy-loading modules)
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37682 Bug ID: 37682 Summary: Improve speed of koha-preferences CLI tool (by lazy-loading modules) Change sponsored?: --- Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Architecture, internals, and plumbing Assignee: koha-bugs@lists.koha-community.org Reporter: dcook@prosentient.com.au QA Contact: testopia@bugs.koha-community.org By lazy loading C4::/Koha:: modules in the koha-preferences tool, so that they're only used by setter commands, we're able to gain enormous speed improvements for the getter commands. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37682 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |37657 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37657 [Bug 37657] Improve speed of koha-preferences CLI tool (by using minimal dbh) -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37682 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |dcook@prosentient.com.au |ity.org | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37682 --- Comment #1 from David Cook <dcook@prosentient.com.au> --- Created attachment 170491 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=170491&action=edit Bug 37682: Lazy-load modules for setters in koha-preferences tool This change lazy-loads modules for setters in the koha-preferences tool, so that getters are free to run super fast. Between BZ 37657 and BZ 37682 we effectively eliminate the overhead of running "get" or "dump" commands via the koha-preferences tool. Test plan: 1. time misc/admin/koha-preferences get SearchEngine 2. Note time is about .35 seconds 3. time misc/admin/koha-preferences dump 4. Note time is about .35 seconds 5. Create sysprefs.yml --- marcflavour: MARC21 viewMARC: 1 6. time misc/admin/koha-preferences load -i sysprefs.yml 7. Note time is about .35 seconds 8. time misc/admin/koha-preferences set SearchEngine Elasticsearch 9. Note time is about 1 seconds 10. Apply patch 11. Repeat the koha-preferences commands above 12. Note that the "dump" and "get" commands run in about .09-.1 seconds. The "load" and "set" commands still take the same amount of time as their behaviours haven't changed 13. misc/admin/koha-preferences set SearchEngine Elasticsearch1 14. koha-mysql kohadev 15. select * from action_logs where module = 'SYSTEMPREFERENCE' order by action_id desc limit 5; 16. Note that the action log showing Elasticsearch1 update says "interface" of "commandline" and "script" of "koha-preferences" -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37682 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37682 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |Academy -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37682 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #170491|0 |1 is obsolete| | --- Comment #2 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 170521 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=170521&action=edit Bug 37682: Lazy-load modules for setters in koha-preferences tool This change lazy-loads modules for setters in the koha-preferences tool, so that getters are free to run super fast. Between BZ 37657 and BZ 37682 we effectively eliminate the overhead of running "get" or "dump" commands via the koha-preferences tool. Test plan: 1. time misc/admin/koha-preferences get SearchEngine 2. Note time is about .35 seconds 3. time misc/admin/koha-preferences dump 4. Note time is about .35 seconds 5. Create sysprefs.yml Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37682 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #170521|0 |1 is obsolete| | --- Comment #3 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 170522 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=170522&action=edit Bug 37682: Lazy-load modules for setters in koha-preferences tool This change lazy-loads modules for setters in the koha-preferences tool, so that getters are free to run super fast. Between BZ 37657 and BZ 37682 we effectively eliminate the overhead of running "get" or "dump" commands via the koha-preferences tool. Test plan: 1. time misc/admin/koha-preferences get SearchEngine 2. Note time is about .35 seconds 3. time misc/admin/koha-preferences dump 4. Note time is about .35 seconds 5. Create sysprefs.yml --- marcflavour: MARC21 viewMARC: 1 6. time misc/admin/koha-preferences load -i sysprefs.yml 7. Note time is about .35 seconds 8. time misc/admin/koha-preferences set SearchEngine Elasticsearch 9. Note time is about 1 seconds 10. Apply patch 11. Repeat the koha-preferences commands above 12. Note that the "dump" and "get" commands run in about .09-.1 seconds. The "load" and "set" commands still take the same amount of time as their behaviours haven't changed 13. misc/admin/koha-preferences set SearchEngine Elasticsearch1 14. koha-mysql kohadev 15. select * from action_logs where module = 'SYSTEMPREFERENCE' order by action_id desc limit 5; 16. Note that the action log showing Elasticsearch1 update says "interface" of "commandline" and "script" of "koha-preferences" Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37682 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |martin.renvoize@ptfs-europe | |.com Status|Needs Signoff |Signed Off --- Comment #4 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Another one I feel confident going straight for QA on.. all works as described and the speed boost is significant. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37682 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |martin.renvoize@ptfs-europe |y.org |.com Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37682 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |julian.maurice@biblibre.com --- Comment #5 from Julian Maurice <julian.maurice@biblibre.com> --- Note that Koha::Script uses C4::Context, so C4::Context::import is executed twice (once with require Koha::Script, once with C4::Context->import) Koha::Logger::import is also "called" twice (it doesn't exist so it doesn't matter much IMO) And you might want to get rid of C4::Log which does not appear to be useful here. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37682 --- Comment #6 from David Cook <dcook@prosentient.com.au> --- (In reply to Julian Maurice from comment #5)
Note that Koha::Script uses C4::Context, so C4::Context::import is executed twice (once with require Koha::Script, once with C4::Context->import) Koha::Logger::import is also "called" twice (it doesn't exist so it doesn't matter much IMO)
In my testing, "use Koha::Script" followed by "use C4::Context" calls C4::Context->import" twice as well, so same same. I haven't investigated Koha::Logger but better safe than sorry. Again, "use Koha::Logger" does the same thing.
And you might want to get rid of C4::Log which does not appear to be useful here.
I'm just doing a like for like so far. Happy for someone to do a follow up. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37682 --- Comment #7 from David Cook <dcook@prosentient.com.au> --- Note: I'm running this already in production, and it's wonderful. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37682 --- Comment #8 from Julian Maurice <julian.maurice@biblibre.com> --- (In reply to David Cook from comment #7)
Note: I'm running this already in production, and it's wonderful. Just to be clear: I didn't mean to block the patch from being pushed and just wanted to point out what could be improved.
In my testing, "use Koha::Script" followed by "use C4::Context" calls C4::Context->import" twice as well, so same same. IIUC the 2nd call is useless but it's exactly like before the patch, so no
(In reply to David Cook from comment #6) problem here. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37682 --- Comment #9 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to David Cook from comment #7)
Note: I'm running this already in production, and it's wonderful.
Could you explain a bit on how you use it? Just interested in use cases. :) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37682 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |24.11.00 released in| | Status|Passed QA |Pushed to main -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37682 --- Comment #10 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Pushed for 24.11! Well done everyone, thank you! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37682 --- Comment #11 from David Cook <dcook@prosentient.com.au> --- (In reply to Katrin Fischer from comment #9)
(In reply to David Cook from comment #7)
Note: I'm running this already in production, and it's wonderful.
Could you explain a bit on how you use it? Just interested in use cases. :)
It's used by "koha-list --elasticsearch", which is invoked by the "koha-common" service. During a "apt-get install koha-common", it actually gets run twice through the process, so it can save a lot of time for deployments. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37682 --- Comment #12 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to David Cook from comment #11)
(In reply to Katrin Fischer from comment #9)
(In reply to David Cook from comment #7)
Note: I'm running this already in production, and it's wonderful.
Could you explain a bit on how you use it? Just interested in use cases. :)
It's used by "koha-list --elasticsearch", which is invoked by the "koha-common" service.
During a "apt-get install koha-common", it actually gets run twice through the process, so it can save a lot of time for deployments.
I understand, thanks! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37682 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|Academy | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37682 Martin Renvoize (ashimema) <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |release-notes-needed -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37682 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the| |By lazy-loading modules release notes| |needed for some functions | |of the koha-preferences CLI | |tool, other functions which | |do not need those modules | |now run much faster. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37682 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|release-notes-needed | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37682 Bug 37682 depends on bug 37657, which changed state. Bug 37657 Summary: Improve speed of koha-preferences CLI tool (by using minimal dbh) https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37657 What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to main |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37682 Lucas Gass (lukeg) <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED CC| |lucas@bywatersolutions.com Status|Pushed to main |RESOLVED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37682 Martin Renvoize (ashimema) <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |robin@catalyst.net.nz Component|Architecture, internals, |Command-line Utilities |and plumbing | -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org