[Bug 19819] New: C4::Context->dbh is unreasonably slow
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19819 Bug ID: 19819 Summary: C4::Context->dbh is unreasonably slow Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Architecture, internals, and plumbing Assignee: koha-bugs@lists.koha-community.org Reporter: glasklas@gmail.com QA Contact: testopia@bugs.koha-community.org I have been profiling Koha the last day or so to try to fix some bottlenecks increasing performance of bulk-processing of data. Discovered that C4::Context->dbh was unexpectedly slow (Koha::Database->schema->storage->dbh is the culprit). If it is safe to do so, performance can be greatly improved by locally caching $dbh in a state variable. Patch will be provided shortly. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19819 --- Comment #1 from David Gustafsson <glasklas@gmail.com> --- Created attachment 69822 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=69822&action=edit Bug 19819: C4::Context->dbh is unreasonably slow Locally cache Koha::Database->schema->storage->dbh in state variable in C4::Context->dbh to get rid of retrieval overhead -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19819 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |glasklas@gmail.com |ity.org | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19819 Josef Moravec <josef.moravec@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |josef.moravec@gmail.com --- Comment #2 from Josef Moravec <josef.moravec@gmail.com> --- Is this ready to test? What should the test plan looks like? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19819 --- Comment #3 from David Gustafsson <glasklas@gmail.com> --- Yes, pretty hard to write a test plan for this. But I guess I can post a script on monday which can be run to illustrate the performance differance. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19819 Mason James <mtj@kohaaloha.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mtj@kohaaloha.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19819 --- Comment #4 from David Gustafsson <glasklas@gmail.com> --- Regarding how to test: time this script (for example) https://gist.github.com/gnucifer/8a9a26de02d2e344c99398a19fc8df71 in koha-shell wihtout the patch applied: `koha-shell kohadev -c "time perl context_dbh_text.pl` On my machine this takes about 17.2 seconds. Apply patch, run again. For me with patch applied time decreases to about 10.2 seconds. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19819 --- Comment #5 from David Gustafsson <glasklas@gmail.com> --- Command should be: `koha-shell <koha_instance_name> -c "time perl context_dbh_text.pl"` -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19819 --- Comment #6 from David Gustafsson <glasklas@gmail.com> --- `koha-shell <koha_instance_name> -c "time perl context_dbh_test.pl"` -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19819 --- Comment #7 from Josef Moravec <josef.moravec@gmail.com> --- My times: Before: real 0m16.295s user 0m11.076s sys 0m1.656s After patch: real 0m8.858s user 0m6.044s sys 0m0.824s -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19819 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19819 Josef Moravec <josef.moravec@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19819 Josef Moravec <josef.moravec@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #69822|0 |1 is obsolete| | --- Comment #8 from Josef Moravec <josef.moravec@gmail.com> --- Created attachment 70249 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=70249&action=edit Bug 19819: C4::Context->dbh is unreasonably slow Locally cache Koha::Database->schema->storage->dbh in state variable in C4::Context->dbh to get rid of retrieval overhead Signed-off-by: Josef Moravec <josef.moravec@gmail.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19819 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@bugs.koha-c | |ommunity.org --- Comment #9 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Did you try this patch in production? I am expecting regressions on the issues with had on bug 14778 (like "server has gone away"). -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19819 --- Comment #10 from David Gustafsson <glasklas@gmail.com> --- Good catch, I have only run it locally, but it will be included the next release for us. I'm not very knowledge about the inner working of Plack, but if the connection is persisted over time, and never closed, I understand how this can be an issue. I could try too look into this some more trying to figure out if the current fix if safe, and if not if there is some other way of going about this. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19819 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |julian.maurice@biblibre.com Status|Signed Off |Failed QA --- Comment #11 from Julian Maurice <julian.maurice@biblibre.com> --- I can confirm that with this patch, if I restart mysql server, koha is no longer able to connect to mysql until I restart (or reload) starman. Looking at the test script and times, this seems to save ~40μs by call (~8s / 200000 calls). That's not much. Do we know how many times by page dbh get called ? Changing status to Failed QA. The patch introduces a regression and I think we need more precise profiling data before changing the implementation of dbh. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19819 --- Comment #12 from David Gustafsson <glasklas@gmail.com> --- It actually has an significant impact on indexing (about 5 - 10% of execution time I think), but I have not had the time to think of a proper fix. I probably will some time in the near future, but I won't protest if someone wants to close the issue. I can open a new one when have a proper fix in that case. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19819 --- Comment #13 from Julian Maurice <julian.maurice@biblibre.com> --- (In reply to David Gustafsson from comment #12)
It actually has an significant impact on indexing (about 5 - 10% of execution time I think), but I have not had the time to think of a proper fix. I probably will some time in the near future, but I won't protest if someone wants to close the issue. I can open a new one when have a proper fix in that case.
No one is going to close the bug report, but the current patch cannot be accepted as it is. The bug report will stay open until you (or someone else) can provide a proper fix. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19819 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au --- Comment #14 from David Cook <dcook@prosentient.com.au> --- (In reply to Jonathan Druart from comment #9)
Did you try this patch in production? I am expecting regressions on the issues with had on bug 14778 (like "server has gone away").
This was exactly what I was going to say. Database connection caching is a nice idea but not something you want to implement naively. I'd suggest not trying to re-invent the wheel on this one, and look somewhere else for database connection pooling. And actually looking at C4::Context::dbh... I see that Koha::Database->schema->storage->dbh... which does cache the $schema object in a package level variable. This bug report was opened nearly 2 years ago, so maybe it is no longer relevant. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19819 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |RESOLVED Resolution|--- |INVALID --- Comment #15 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Agreed. The cache is done already, I am not sure we can do better here. I am closing it for now, but feel free to reopen if you have something to suggest. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19819 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|INVALID |DUPLICATE --- Comment #16 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- I forgot about bug 16156, it has a patch that was waiting for feedback. The idea was similar, but $dbh cache would be flushed when needed. *** This bug has been marked as a duplicate of bug 16156 *** -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon@bugs.koha-community.org