[Bug 7276] New: member entry Performance improvement
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7276 Bug #: 7276 Summary: member entry Performance improvement Classification: Unclassified Change sponsored?: --- Product: Koha Version: rel_3_6 Platform: All OS/Version: All Status: NEW Severity: normal Priority: P5 - low Component: Architecture, internals, and plumbing AssignedTo: gmcharlt@gmail.com ReportedBy: stephane.delaune@biblibre.com QAContact: ian.walls@bywatersolutions.com Before this patch : 9619 borrowers added in 31 Minutes After : 68 seconds. -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7276 --- Comment #1 from stephane.delaune@biblibre.com 2011-11-29 10:18:21 UTC --- Created attachment 6453 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=6453 the csv file containing anonymised borrowers to import -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7276 stephane.delaune@biblibre.com changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P5 - low |P2 -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7276 --- Comment #2 from stephane.delaune@biblibre.com 2011-11-29 10:36:10 UTC --- Created attachment 6454 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=6454 proposed patch -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7276 Paul Poulain <paul.poulain@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P2 |PATCH-Sent Status|NEW |ASSIGNED CC| |paul.poulain@biblibre.com Patch Status|--- |Needs Signoff AssignedTo|gmcharlt@gmail.com |stephane.delaune@biblibre.c | |om -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7276 stephane.delaune@biblibre.com changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|PATCH-Sent |P2 --- Comment #3 from stephane.delaune@biblibre.com 2011-12-02 11:27:42 UTC --- The fix change the addmember sub, add a borrower in koha to test it -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7276 Ian Walls <ian.walls@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P2 |PATCH-Sent --- Comment #4 from Ian Walls <ian.walls@bywatersolutions.com> 2011-12-02 14:21:30 UTC --- resetting priority so Patch Status will show up. Very encouraging looking patch. Please test thoroughly in not only Member Entry, but also Member search and acquisitions, as this change may affect those areas as well. -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7276 --- Comment #5 from Paul Poulain <paul.poulain@biblibre.com> 2011-12-05 20:47:45 UTC --- This patch show a big improvement in performance when we bulk load patrons, using a specific script we're using at BibLibre when doing our migrations. I think the improvement can be seen in tools > import patrons, when loading large sets of patrons. Note this patch is a porting of what we've made in git.biblibre.com and use in productions for months (if not years) -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7276 Chris Cormack <chris@bigballofwax.co.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |chris@bigballofwax.co.nz --- Comment #6 from Chris Cormack <chris@bigballofwax.co.nz> 2011-12-06 21:58:50 UTC --- This patch looks good, but as we are moving to a persistent running environment for Koha, we need to make sure we have a way to clear the table structure from the variable. So I will submit a follow up to clear the variable, which can be called by the update scripts when it changes the table structure. Similar to clear_syspref_cache in C4::Context This patch does give a huge improvement in patron import though, so I am going to sign off and send that follow up. -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7276 Chris Cormack <chris@bigballofwax.co.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #6454|0 |1 is obsolete| | --- Comment #7 from Chris Cormack <chris@bigballofwax.co.nz> 2011-12-06 22:03:01 UTC --- Created attachment 6623 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=6623 Bug 7276 : member entry Performance improvement Before this patch : 9619 borrowers added in 31 Minutes, After : 68 seconds. This adds Hashref to table structure in C4::SQLHelper to speed up bulk edits. Signed-off-by: Stéphane Delaune <stephane.delaune@biblibre.com> Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz> -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7276 --- Comment #8 from Chris Cormack <chris@bigballofwax.co.nz> 2011-12-06 22:08:07 UTC --- Created attachment 6624 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=6624 Bug 7276 : Follow up, adding a sub to clear the cache -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7276 --- Comment #9 from Chris Cormack <chris@bigballofwax.co.nz> 2011-12-06 22:08:33 UTC --- Only the follow up needs sign off -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7276 --- Comment #10 from Paul Poulain <paul.poulain@biblibre.com> 2011-12-09 15:12:50 UTC --- QA comment about the follow-up = the initial patch caches the database structure for up to 10 minuts. It means the clearing sub is not necessary imo, even under a persistent env, that will work. It will just be needed to wait for up to 10 minuts to get a fresh env. Chris, can you confirm you've seen the 10 minuts limits, and explain why it should not be enough ? (follow-up OK and everything passed QA though) -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7276 --- Comment #11 from Chris Cormack <chris@bigballofwax.co.nz> 2011-12-09 16:55:11 UTC --- (In reply to comment #10)
QA comment
about the follow-up = the initial patch caches the database structure for up to 10 minuts. It means the clearing sub is not necessary imo, even under a persistent env, that will work. It will just be needed to wait for up to 10 minuts to get a fresh env.
Chris, can you confirm you've seen the 10 minuts limits, and explain why it should not be enough ?
(follow-up OK and everything passed QA though)
We are moving slowly to a persistent model, be it mod_perl, plack, fastcgi ..whatever. As such variables declared like this hash need to be able to be wiped. This has nothing to do with the memcached caching, but that we are declaring a variable, that when we are running in persistent mode will persist. So, we should always clean up after ourselves or we will get inconsistent and wrong behaviour when running in a persistent mode. -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7276 --- Comment #12 from Paul Poulain <paul.poulain@biblibre.com> 2011-12-09 17:12:46 UTC --- OK, I undestand what I missed: cleaning the variable is necessary, otherwise, it will stay forever in your variable, which is bad. It has nothing to do with memcache PS: maybe we should write guidelines to manage persistent variables PS2: with a persistent model, will memcache still be usefull ? (i'll push the patch on monday) -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7276 --- Comment #13 from Chris Cormack <chris@bigballofwax.co.nz> 2011-12-10 09:13:33 UTC --- (In reply to comment #12)
OK, I undestand what I missed: cleaning the variable is necessary, otherwise, it will stay forever in your variable, which is bad. It has nothing to do with memcache
PS: maybe we should write guidelines to manage persistent variables
Probably a good idea.
PS2: with a persistent model, will memcache still be usefull ?
Yes, even with persistence, threads die after a while and respawn. So it is still useful. -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7276 Chris Cormack <chris@bigballofwax.co.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch Status|Needs Signoff |Signed Off -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7276 D Ruth Bavousett <ruth@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ruth@bywatersolutions.com -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7276 Paul Poulain <paul.poulain@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7276 Paul Poulain <paul.poulain@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to Master -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7276 --- Comment #14 from Chris Cormack <chris@bigballofwax.co.nz> 2012-01-17 19:10:23 UTC --- (In reply to comment #13)
(In reply to comment #12)
OK, I undestand what I missed: cleaning the variable is necessary, otherwise, it will stay forever in your variable, which is bad. It has nothing to do with memcache
PS: maybe we should write guidelines to manage persistent variables
Probably a good idea.
PS2: with a persistent model, will memcache still be usefull ?
Yes, even with persistence, threads die after a while and respawn. So it is still useful.
Following up, FWIW on some of huge traffic sites (not Koha sites but other sites) we run with persistance, and memcached. When you move to a cluster model this really becomes useful, but even single server, removing a lot of unnessecary reads from your db helps it do writes much faster. -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7276 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to Master |Pushed to Stable CC| |katrin.fischer@bsz-bw.de -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7276 Chris Cormack <chris@bigballofwax.co.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to Stable |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org