[Bug 37657] New: koha-list --elasticsearch is super slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37657 Bug ID: 37657 Summary: koha-list --elasticsearch is super slow Change sponsored?: --- Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Architecture, internals, and plumbing Assignee: koha-bugs@lists.koha-community.org Reporter: dcook@prosentient.com.au QA Contact: testopia@bugs.koha-community.org If you have a large number of instances on a server, "koha-list --elasticsearch" takes a very long time to run. I think it's about 1-2 seconds per instance. That probably doesn't sound like much, but if you multiple it against a large number then it starts adding up. If it's run as part of an "apt-get install koha-common", that means that you've got a lot of downtime for what should be a very simple task. It's not an urgent issue, but I figure at some point I'll look at optimizing this lookup. The fun part with apt-get install koha-common is that "koha-list --elasticsearch" gets run 2 times. On some test machines I'm running, the slowest part of an update is "koha-list --elasticsearch". -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37657 --- Comment #1 from David Cook <dcook@prosentient.com.au> --- If I use Koha::Database->dbh instead of C4::Context->dbh, I can get "misc/admin/koha-preferences get SearchEngine" down from 0.914s to 0.328s. That's pretty impressive. However, if I remove the other Koha::/C4:: module loads from the top of the script, I can reduce it even further down to 0.087s. That's the kind of speed I'm really looking for here. -- I reckon lazy-loading the modules needed for each command isn't a bad way to go. Even if the community doesn't accept it, I'm going to do it locally anyway, as the speed improvement is just too good not to do. But perhaps I'll use this bug report to do that first speed improvement, and then open a separate one to do the lazy-loading to get the final speed improvement, since that one will be more controversial. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37657 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=34088 -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37657 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |dcook@prosentient.com.au |ity.org | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37657 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|koha-list --elasticsearch |Improve speed of |is super slow |koha-preferences CLI tool -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37657 --- Comment #2 from David Cook <dcook@prosentient.com.au> --- Created attachment 170490 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=170490&action=edit Bug 37657: Make koha-preferences CLI tool faster By using Koha::Database->dbh() to use a minimal database handle which doesn't preload the whole DBIx::Class schema, we're able to run the same command 2-3 times faster. This is beneficial when running the tool in a loop which runs the command serially one by one. Test plan: 1. time misc/admin/koha-preferences get SearchEngine 2. Note time is about 1 second 3. time misc/admin/koha-preferences dump 4. Note time is about 1 second 5. Create sysprefs.yml --- marcflavour: MARC21 viewMARC: 1 6. time misc/admin/koha-preferences load -i sysprefs.yml 7. Apply patch 8. Repeat the koha-preferences commands above 9. Note that the command runs about 2-3 times faster -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37657 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Improve speed of |Improve speed of |koha-preferences CLI tool |koha-preferences CLI tool | |(by using minimal dbh) Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37657 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |37682 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37682 [Bug 37682] Improve speed of koha-preferences CLI tool (by lazy-loading modules) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37657 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |Academy -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37657 --- Comment #3 from David Cook <dcook@prosentient.com.au> --- Note: I think the only place that "misc/admin/koha-preferences" is used directly by Koha is in debian/scripts/koha-functions.sh where it is used in the function "is_elasticsearch_enabled". So I think this is a very low risk change as well. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37657 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #170490|0 |1 is obsolete| | --- Comment #4 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 170520 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=170520&action=edit Bug 37657: Make koha-preferences CLI tool faster By using Koha::Database->dbh() to use a minimal database handle which doesn't preload the whole DBIx::Class schema, we're able to run the same command 2-3 times faster. This is beneficial when running the tool in a loop which runs the command serially one by one. Test plan: 1. time misc/admin/koha-preferences get SearchEngine 2. Note time is about 1 second 3. time misc/admin/koha-preferences dump 4. Note time is about 1 second 5. Create sysprefs.yml Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37657 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |martin.renvoize@ptfs-europe | |.com Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37657 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |martin.renvoize@ptfs-europe |y.org |.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37657 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA --- Comment #5 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Trivial, clear, meaningful patch that certainly reduced memory footprint and increased speed for these scripts. Going straight for Passing QA. I don't entirely understand why C4::Context->dbh isn't just a passthrough for Koha::Database->dbh at this point.. I can't really see why you'd want a Schema loaded when you're clearly only asking for the database handle back... one for another bug though that. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37657 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to main Version(s)| |24.11.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37657 --- Comment #6 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Pushed for 24.11! Well done everyone, thank you! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37657 Martin Renvoize (ashimema) <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |release-notes-needed -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37657 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|release-notes-needed | Text to go in the| |This change refactors the release notes| |koha-preferences CLI tool | |to use different internal | |database libraries in order | |to gain a speed performance | |improvement. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37657 Lucas Gass (lukeg) <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED CC| |lucas@bywatersolutions.com Status|Pushed to main |RESOLVED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37657 Martin Renvoize (ashimema) <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Component|Architecture, internals, |Command-line Utilities |and plumbing | CC| |robin@catalyst.net.nz -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org